Online Access Free Associate-Developer-Apache-Spark-3.5 Practice Test

A Spark application developer wants to identify which operations cause shuffling, leading to a new stage in the Spark execution plan.
Which operation results in a shuffle and a new stage?

• A.DataFrame.select()
• B.DataFrame.groupBy().agg()
• C.DataFrame.withColumn()
• D.DataFrame.filter()

A data engineer is reviewing a Spark application that applies several transformations to a DataFrame but notices that the job does not start executing immediately.
Which two characteristics of Apache Spark's execution model explain this behavior?
Choose 2 answers:

• A.Transformations are evaluated lazily.
• B.Transformations are executed immediately to build the lineage graph.
• C.The Spark engine optimizes the execution plan during the transformations, causing delays.
• D.Only actions trigger the execution of the transformation pipeline.
• E.The Spark engine requires manual intervention to start executing transformations.

A data engineer is working on the DataFrame:

(Referring to the table image: it has columns Id, Name, count, and timestamp.) Which code fragment should the engineer use to extract the unique values in the Name column into an alphabetically ordered list?

• A.df.select("Name").distinct().orderBy(df["Name"].desc())
• B.df.select("Name").distinct()
• C.df.select("Name").orderBy(df["Name"].asc())
• D.df.select("Name").distinct().orderBy(df["Name"])

18 of 55.
An engineer has two DataFrames - df1 (small) and df2 (large). To optimize the join, the engineer uses a broadcast join:
from pyspark.sql.functions import broadcast
df_result = df2.join(broadcast(df1), on="id", how="inner")
What is the purpose of using broadcast() in this scenario?

• A.It increases the partition size for df1 and df2.
• B.It reduces the number of shuffle operations by replicating the smaller DataFrame to all nodes.
• C.It filters the id values before performing the join.
• D.It ensures that the join happens only when the id values are identical.

33 of 55.
The data engineering team created a pipeline that extracts data from a transaction system.
The transaction system stores timestamps in UTC, and the data engineers must now transform the transaction_datetime field to the "America/New_York" timezone for reporting.
Which code should be used to convert the timestamp to the target timezone?

• A.raw.withColumn("transaction_datetime", to_utc_timestamp(col("transaction_datetime"), "America/New_York"))
• B.raw.withColumn("transaction_datetime", date_format(col("transaction_datetime"), "America/New_York"))
• C.raw.withColumn("transaction_datetime", from_utc_timestamp(col("transaction_datetime"), "America/New_York"))
• D.raw.withColumn("transaction_datetime", convert_timezone(col("transaction_datetime"), "America/New_York"))