Online Access Free Associate-Developer-Apache-Spark-3.5 Practice Test

A data engineer is asked to build an ingestion pipeline for a set of Parquet files delivered by an upstream team on a nightly basis. The data is stored in a directory structure with a base path of "/path/events/data". The upstream team drops daily data into the underlying subdirectories following the convention year/month/day.
A few examples of the directory structure are:

Which of the following code snippets will read all the data within the directory structure?

• A.df = spark.read.option("inferSchema", "true").parquet("/path/events/data/")
• B.df = spark.read.parquet("/path/events/data/")
• C.df = spark.read.parquet("/path/events/data/*")
• D.df = spark.read.option("recursiveFileLookup", "true").parquet("/path/events/data/")

A Data Analyst is working on the DataFrame sensor_df, which contains two columns:
Which code fragment returns a DataFrame that splits the record column into separate columns and has one array item per row?
A)

B)

C)

D)

• A.exploded_df = exploded_df.select(
  "record_datetime",
  "record_exploded.sensor_id",
  "record_exploded.status",
  "record_exploded.health"
  )
  exploded_df = sensor_df.withColumn("record_exploded", explode("record"))
• B.exploded_df = exploded_df.select(
  "record_datetime",
  "record_exploded.sensor_id",
  "record_exploded.status",
  "record_exploded.health"
  )
  exploded_df = sensor_df.withColumn("record_exploded", explode("record"))
• C.exploded_df = exploded_df.select("record_datetime", "record_exploded")
• D.exploded_df = sensor_df.withColumn("record_exploded", explode("record")) exploded_df = exploded_df.select("record_datetime", "sensor_id", "status", "health")

33 of 55.
The data engineering team created a pipeline that extracts data from a transaction system.
The transaction system stores timestamps in UTC, and the data engineers must now transform the transaction_datetime field to the "America/New_York" timezone for reporting.
Which code should be used to convert the timestamp to the target timezone?

• A.raw.withColumn("transaction_datetime", to_utc_timestamp(col("transaction_datetime"), "America/New_York"))
• B.raw.withColumn("transaction_datetime", convert_timezone(col("transaction_datetime"), "America/New_York"))
• C.raw.withColumn("transaction_datetime", date_format(col("transaction_datetime"), "America/New_York"))
• D.raw.withColumn("transaction_datetime", from_utc_timestamp(col("transaction_datetime"), "America/New_York"))

18 of 55.
An engineer has two DataFrames - df1 (small) and df2 (large). To optimize the join, the engineer uses a broadcast join:
from pyspark.sql.functions import broadcast
df_result = df2.join(broadcast(df1), on="id", how="inner")
What is the purpose of using broadcast() in this scenario?

• A.It ensures that the join happens only when the id values are identical.
• B.It filters the id values before performing the join.
• C.It reduces the number of shuffle operations by replicating the smaller DataFrame to all nodes.
• D.It increases the partition size for df1 and df2.

39 of 55.
A Spark developer is developing a Spark application to monitor task performance across a cluster.
One requirement is to track the maximum processing time for tasks on each worker node and consolidate this information on the driver for further analysis.
Which technique should the developer use?

• A.Broadcast a variable to share the maximum time among workers.
• B.Use an RDD action like reduce() to compute the maximum time.
• C.Configure the Spark UI to automatically collect maximum times.
• D.Use an accumulator to record the maximum time on the driver.