The code block displayed below contains an error. The code block should merge the rows of DataFrames transactionsDfMonday and transactionsDfTuesday into a new DataFrame, matching column names and inserting null values where column names do not appear in both DataFrames. Find the error. Sample of DataFrame transactionsDfMonday: 1.+-------------+---------+-----+-------+---------+----+ 2.|transactionId|predError|value|storeId|productId| f| 3.+-------------+---------+-----+-------+---------+----+ 4.| 5| null| null| null| 2|null| 5.| 6| 3| 2| 25| 2|null| 6.+-------------+---------+-----+-------+---------+----+ Sample of DataFrame transactionsDfTuesday: 1.+-------+-------------+---------+-----+ 2.|storeId|transactionId|productId|value| 3.+-------+-------------+---------+-----+ 4.| 25| 1| 1| 4| 5.| 2| 2| 2| 7| 6.| 3| 4| 2| null| 7.| null| 5| 2| null| 8.+-------+-------------+---------+-----+ Code block: sc.union([transactionsDfMonday, transactionsDfTuesday])
Correct Answer: E
Explanation Correct code block: transactionsDfMonday.unionByName(transactionsDfTuesday, True) Output of correct code block: +-------------+---------+-----+-------+---------+----+ |transactionId|predError|value|storeId|productId| f| +-------------+---------+-----+-------+---------+----+ | 5| null| null| null| 2|null| | 6| 3| 2| 25| 2|null| | 1| null| 4| 25| 1|null| | 2| null| 7| 2| 2|null| | 4| null| null| 3| 2|null| | 5| null| null| null| 2|null| +-------------+---------+-----+-------+---------+----+ For solving this question, you should be aware of the difference between the DataFrame.union() and DataFrame.unionByName() methods. The first one matches columns independent of their names, just by their order. The second one matches columns by their name (which is asked for in the question). It also has a useful optional argument, allowMissingColumns. This allows you to merge DataFrames that have different columns - just like in this example. sc stands for SparkContext and is automatically provided when executing code on Databricks. While sc.union() allows you to join RDDs, it is not the right choice for joining DataFrames. A hint away from sc.union() is given where the question talks about joining "into a new DataFrame". concat is a method in pyspark.sql.functions. It is great for consolidating values from different columns, but has no place when trying to join rows of multiple DataFrames. Finally, the join method is a contender here. However, the default join defined for that method is an inner join which does not get us closer to the goal to match the two DataFrames as instructed, especially given that with the default arguments we cannot define a join condition. More info: - pyspark.sql.DataFrame.unionByName - PySpark 3.1.2 documentation - pyspark.SparkContext.union - PySpark 3.1.2 documentation - pyspark.sql.functions.concat - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
Question 42
Which of the following code blocks generally causes a great amount of network traffic?
Correct Answer: C
Explanation DataFrame.collect() sends all data in a DataFrame from executors to the driver, so this generally causes a great amount of network traffic in comparison to the other options listed. DataFrame.coalesce() just reduces the number of partitions and generally aims to reduce network traffic in comparison to a full shuffle. DataFrame.select() is evaluated lazily and, unless followed by an action, does not cause significant network traffic. DataFrame.rdd.map() is evaluated lazily, it does therefore not cause great amounts of network traffic. DataFrame.count() is an action. While it does cause some network traffic, for the same DataFrame, collecting all data in the driver would generally be considered to cause a greater amount of network traffic.
Question 43
The code block shown below should return a one-column DataFrame where the column storeId is converted to string type. Choose the answer that correctly fills the blanks in the code block to accomplish this. transactionsDf.__1__(__2__.__3__(__4__))
Correct Answer: D
Explanation Correct code block: transactionsDf.select(col("storeId").cast(StringType())) Solving this question involves understanding that, when using types from the pyspark.sql.types such as StringType, these types need to be instantiated when using them in Spark, or, in simple words, they need to be followed by parentheses like so: StringType(). You could also use .cast("string") instead, but that option is not given here. More info: pyspark.sql.Column.cast - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2
Question 44
Which of the following code blocks performs an inner join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively, excluding columns value and storeId from DataFrame transactionsDf and column attributes from DataFrame itemsDf?
Correct Answer: E
Explanation This question offers you a wide variety of answers for a seemingly simple question. However, this variety reflects the variety of ways that one can express a join in PySpark. You need to understand some SQL syntax to get to the correct answer here. transactionsDf.createOrReplaceTempView('transactionsDf') itemsDf.createOrReplaceTempView('itemsDf') statement = """ SELECT * FROM transactionsDf INNER JOIN itemsDf ON transactionsDf.productId==itemsDf.itemId """ spark.sql(statement).drop("value", "storeId", "attributes") Correct - this answer uses SQL correctly to perform the inner join and afterwards drops the unwanted columns. This is totally fine. If you are unfamiliar with the triple-quote """ in Python: This allows you to express strings as multiple lines. transactionsDf \ drop(col('value'), col('storeId')) \ join(itemsDf.drop(col('attributes')), col('productId')==col('itemId')) No, this answer option is a trap, since DataFrame.drop() does not accept a list of Column objects. You could use transactionsDf.drop('value', 'storeId') instead. transactionsDf.drop("value", "storeId").join(itemsDf.drop("attributes"), "transactionsDf.productId==itemsDf.itemId") Incorrect - Spark does not evaluate "transactionsDf.productId==itemsDf.itemId" as a valid join expression. This would work if it would not be a string. transactionsDf.drop('value', 'storeId').join(itemsDf.select('attributes'), transactionsDf.productId==itemsDf.itemId) Wrong, this statement incorrectly uses itemsDf.select instead of itemsDf.drop. transactionsDf.createOrReplaceTempView('transactionsDf') itemsDf.createOrReplaceTempView('itemsDf') spark.sql("SELECT -value, -storeId FROM transactionsDf INNER JOIN itemsDf ON productId==itemId").drop("attributes") No, here the SQL expression syntax is incorrect. Simply specifying -columnName does not drop a column. More info: pyspark.sql.DataFrame.join - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
Question 45
Which of the following describes tasks?
Correct Answer: E
Explanation Tasks get assigned to the executors by the driver. Correct! Or, in other words: Executors take the tasks that they were assigned to by the driver, run them over partitions, and report the their outcomes back to the driver. Tasks transform jobs into DAGs. No, this statement disrespects the order of elements in the Spark hierarchy. The Spark driver transforms jobs into DAGs. Each job consists of one or more stages. Each stage contains one or more tasks. A task is a collection of rows. Wrong. A partition is a collection of rows. Tasks have little to do with a collection of rows. If anything, a task processes a specific partition. A task is a command sent from the driver to the executors in response to a transformation. Incorrect. The Spark driver does not send anything to the executors in response to a transformation, since transformations are evaluated lazily. So, the Spark driver would send tasks to executors only in response to actions. A task is a collection of slots. No. Executors have one or more slots to process tasks and each slot can be assigned a task.
