Online Access Free Associate-Developer-Apache-Spark-3.5 Practice Test

You have:
DataFrame A: 128 GB of transactions
DataFrame B: 1 GB user lookup table
Which strategy is correct for broadcasting?

• A.DataFrame A should be broadcasted because it is smaller and will eliminate the need for shuffling itself
• B.DataFrame B should be broadcasted because it is smaller and will eliminate the need for shuffling itself
• C.DataFrame A should be broadcasted because it is larger and will eliminate the need for shuffling DataFrame B
• D.DataFrame B should be broadcasted because it is smaller and will eliminate the need for shuffling DataFrame A

A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off.
How can this be achieved?

• A.By configuring the option recoveryLocation during writeStream
• B.By configuring the option checkpointLocation during readStream
• C.By configuring the option checkpointLocation during writeStream
• D.By configuring the option recoveryLocation during the SparkSession initialization

1 of 55. A data scientist wants to ingest a directory full of plain text files so that each record in the output DataFrame contains the entire contents of a single file and the full path of the file the text was read from.
The first attempt does read the text files, but each record contains a single line. This code is shown below:
txt_path = "/datasets/raw_txt/*"
df = spark.read.text(txt_path) # one row per line by default
df = df.withColumn("file_path", input_file_name()) # add full path
Which code change can be implemented in a DataFrame that meets the data scientist's requirements?

• A.Add the option lineSep=", " to the text() function.
• B.Add the option wholetext to the text() function.
• C.Add the option wholetext=False to the text() function.
• D.Add the option lineSep to the text() function.

54 of 55.
What is the benefit of Adaptive Query Execution (AQE)?

• A.It allows Spark to optimize the query plan before execution but does not adapt during runtime.
• B.It automatically distributes tasks across nodes in the clusters and does not perform runtime adjustments to the query plan.
• C.It optimizes query execution by parallelizing tasks and does not adjust strategies based on runtime metrics like data skew.
• D.It enables the adjustment of the query plan during runtime, handling skewed data, optimizing join strategies, and improving overall query performance.

A Data Analyst is working on the DataFrame sensor_df, which contains two columns:
Which code fragment returns a DataFrame that splits the record column into separate columns and has one array item per row?
A)

B)

C)

D)

• A.exploded_df = exploded_df.select(
  "record_datetime",
  "record_exploded.sensor_id",
  "record_exploded.status",
  "record_exploded.health"
  )
  exploded_df = sensor_df.withColumn("record_exploded", explode("record"))
• B.exploded_df = sensor_df.withColumn("record_exploded", explode("record")) exploded_df = exploded_df.select("record_datetime", "sensor_id", "status", "health")
• C.exploded_df = exploded_df.select(
  "record_datetime",
  "record_exploded.sensor_id",
  "record_exploded.status",
  "record_exploded.health"
  )
  exploded_df = sensor_df.withColumn("record_exploded", explode("record"))
• D.exploded_df = exploded_df.select("record_datetime", "record_exploded")