Online Access Free Databricks-Certified-Professional-Data-Engineer Practice Test

A data engineer is tasked with ensuring that a Delta table in Databricks continuously retains deleted files for
15 days instead of the default 7 days, in order to comply with the organization's data retention policy. Which code snippet correctly sets this retention period for deleted files?

• A.spark.sql( " " "
  ALTER TABLE my_table
  SET TBLPROPERTIES ( ' delta.deletedFileRetentionDuration ' = ' interval 15 days ' )
  " " " )
• B.from delta.tables import *
  deltaTable = DeltaTable.forPath(spark, " /mnt/data/my_table " )
  deltaTable.deletedFileRetentionDuration = " interval 15 days "
• C.spark.sql( " VACUUM my_table RETAIN 15 HOURS " )
• D.spark.conf.set( " spark.databricks.delta.deletedFileRetentionDuration " , " 15 days " )

A table in the Lakehouse named customer_churn_params is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours.
Which approach would simplify the identification of these changed records?

• A.Replace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.
• B.Modify the overwrite logic to include a field populated by calling spark.sql.functions.
  current_timestamp() as data are being written; use this field to identify records written on a particular date.
• C.Calculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.
• D.Apply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.
• E.Convert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.

Assuming that the Databricks CLI has been installed and configured correctly, which Databricks CLI command can be used to upload a custom Python Wheel to object storage mounted with the DBFS for use with a production job?

• A.libraries
• B.fs
• C.jobs
• D.configure
• E.workspace

A data engineer is designing a Lakeflow Spark Declarative Pipeline to process streaming order data. The pipeline uses Auto Loader to ingest data and must enforce data quality by ensuring customer_id is not null and amount is greater than zero. Invalid records should be dropped. Which Lakeflow Spark Declarative Pipelines configuration implements this requirement using Python?

• A.@dlt.table
  def silver_orders():
  return dlt.read_stream( " bronze_orders " ) \
  .expect_or_drop( " valid_customer " , " customer_id IS NOT NULL " ) \
  .expect_or_drop( " valid_amount " , " amount > 0 " )
• B.@dlt.table
  def silver_orders():
  return dlt.read_stream( " bronze_orders " ) \
  .expect( " valid_customer " , " customer_id IS NOT NULL " ) \
  .expect( " valid_amount " , " amount > 0 " )
• C.@dlt.table
  @dlt.expect( " valid_customer " , " customer_id IS NOT NULL " )
  @dlt.expect( " valid_amount " , " amount > 0 " )
  def silver_orders():
  return dlt.read_stream( " bronze_orders " )
• D.@dlt.table
  @dlt.expect_or_drop( " valid_customer " , " customer_id IS NOT NULL " )
  @dlt.expect_or_drop( " valid_amount " , " amount > 0 " )
  def silver_orders():
  return dlt.read_stream( " bronze_orders " )

While reviewing a query ' s execution in the Databricks Query Profiler, a data engineer observes that the Top Operators panel shows a Sort operator with high Time Spent and Memory Peak metrics. The Spark UI also reports frequent data spilling .
How should the data engineer address this issue?

• A.Repartition the DataFrame to a single partition before sorting.
• B.Switch to a broadcast join to reduce memory usage.
• C.Convert the sort operation to a filter operation.
• D.Increase the number of shuffle partitions to better distribute data.