Online Access Free Associate-Developer-Apache-Spark-3.5 Practice Test

A developer wants to refactor some older Spark code to leverage built-in functions introduced in Spark 3.5.0. The existing code performs array manipulations manually. Which of the following code snippets utilizes new built-in functions in Spark 3.5.0 for array operations?

• A.
  result_df = prices_df \
  .agg(F.min("spot_price"), F.max("spot_price"))
• B.
  result_df = prices_df \
  .agg(F.count_if(F.col("spot_price") >= F.lit(min_price)))
• C.
  result_df = prices_df \
  .withColumn("valid_price", F.when(F.col("spot_price") > F.lit(min_price), 1).otherwise(0))
• D.
  result_df = prices_df \
  .agg(F.count("spot_price").alias("spot_price")) \
  .filter(F.col("spot_price") > F.lit("min_price"))

41 of 55.
A data engineer is working on the DataFrame df1 and wants the Name with the highest count to appear first (descending order by count), followed by the next highest, and so on.
The DataFrame has columns:
id | Name | count | timestamp
---------------------------------
1 | USA | 10
2 | India | 20
3 | England | 50
4 | India | 50
5 | France | 20
6 | India | 10
7 | USA | 30
8 | USA | 40
Which code fragment should the engineer use to sort the data in the Name and count columns?

• A.df1.sort("Name", "count")
• B.df1.orderBy("Name", "count")
• C.df1.orderBy(col("count").desc(), col("Name").asc())
• D.df1.orderBy(col("Name").desc(), col("count").asc())

A data engineer is building an Apache Spark™ Structured Streaming application to process a stream of JSON events in real time. The engineer wants the application to be fault-tolerant and resume processing from the last successfully processed record in case of a failure. To achieve this, the data engineer decides to implement checkpoints.
Which code snippet should the data engineer use?

• A.query = streaming_df.writeStream \
  .format("console") \
  .outputMode("append") \
  .option("checkpointLocation", "/path/to/checkpoint") \
  .start()
• B.query = streaming_df.writeStream \
  .format("console") \
  .outputMode("append") \
  .start()
• C.query = streaming_df.writeStream \
  .format("console") \
  .option("checkpoint", "/path/to/checkpoint") \
  .outputMode("append") \
  .start()
• D.query = streaming_df.writeStream \
  .format("console") \
  .outputMode("complete") \
  .start()

Given this code:

.withWatermark("event_time", "10 minutes")
.groupBy(window("event_time", "15 minutes"))
.count()
What happens to data that arrives after the watermark threshold?
Options:

• A.Any data arriving more than 10 minutes after the watermark threshold will be ignored and not included in the aggregation.
• B.The watermark ensures that late data arriving within 10 minutes of the latest event_time will be processed and included in the windowed aggregation.
• C.Data arriving more than 10 minutes after the latest watermark will still be included in the aggregation but will be placed into the next window.
• D.Records that arrive later than the watermark threshold (10 minutes) will automatically be included in the aggregation if they fall within the 15-minute window.

22 of 55.
A Spark application needs to read multiple Parquet files from a directory where the files have differing but compatible schemas.
The data engineer wants to create a DataFrame that includes all columns from all files.
Which code should the data engineer use to read the Parquet files and include all columns using Apache Spark?

• A.spark.read.option("mergeSchema", True).parquet("/data/parquet/")
• B.spark.read.parquet("/data/parquet/").option("mergeAllCols", True)
• C.spark.read.parquet("/data/parquet/")
• D.spark.read.format("parquet").option("inferSchema", "true").load("/data/parquet/")