Online Access Free Certified-Data-Engineer-Professional Practice Test

The data science team has created and logged a production model using MLflow. The model accepts a list of column names and returns a new column of type DOUBLE.
The following code correctly imports the production model, loads the customers table containing the customer_id key column into a DataFrame, and defines the feature columns needed for the model.

Which code block will output a DataFrame with the schema "customer_id LONG, predictions DOUBLE"?

• A.df.select("customer_id", model(*columns).alias("predictions"))
• B.df.select("customer_id", pandas_udf(model, columns).alias("predictions"))
• C.model.predict(df, columns)
• D.df.map(lambda x:model(x[columns])).select("customer_id, predictions")
• E.df.apply(model, columns).select("customer_id, predictions")

Which statement characterizes the general programming model used by Spark Structured Streaming?

• A.Structured Streaming leverages the parallel processing of GPUs to achieve highly parallel data throughput.
• B.Structured Streaming models new data arriving in a data stream as new rows appended to an unbounded table.
• C.Structured Streaming is implemented as a messaging bus and is derived from Apache Kafka.
• D.Structured Streaming relies on a distributed network of nodes that hold incremental state values for cached stages.
• E.Structured Streaming uses specialized hardware and I/O streams to achieve sub-second latency for data transfer.

The following table consists of items found in user carts within an e-commerce website.

The following MERGE statement is used to update this table using an updates view, with schema evolution enabled on this table.

How would the following update be handled?

• A.The update throws an error because changes to existing columns in the target schema are not supported.
• B.The update is moved to separate ''restored'' column because it is missing a column expected in the target schema.
• C.The new restored field is added to the target schema, and dynamically read as NULL for existing unmatched records.
• D.The new nested field is added to the target schema, and files underlying existing records are updated to include NULL values for the new field.

A Delta Lake table in the Lakehouse named customer_parsams is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
Immediately after each update succeeds, the data engineer team would like to determine the difference between the new version and the previous of the table. Given the current implementation, which method can be used?

• A.Execute DESCRIBE HISTORY customer_churn_params to obtain the full operation metrics for the update, including a log of all records that have been added or modified.
• B.Parse the Delta Lake transaction log to identify all newly written data files.
• C.Parse the Spark event logs to identify those rows that were updated, inserted, or deleted.
• D.Execute a query to calculate the difference between the new version and the previous version using Delta Lake's built-in versioning and time travel functionality.

A new data engineer notices that a critical field was omitted from an application that writes its Kafka source to Delta Lake. This happened even though the critical field was in the Kafka source.
That field was further missing from data written to dependent, long-term storage. The retention threshold on the Kafka service is seven days. The pipeline has been in production for three months.
Which describes how Delta Lake can help to avoid data loss of this nature in the future?

• A.Ingestine all raw data and metadata from Kafka to a bronze Delta table creates a permanent, replayable history of the data state.
• B.Delta Lake automatically checks that all fields present in the source data are included in the ingestion layer.
• C.Delta Lake schema evolution can retroactively calculate the correct value for newly added fields, as long as the data was in the original source.
• D.Data can never be permanently dropped or deleted from Delta Lake, so data loss is not possible under any circumstance.
• E.The Delta log and Structured Streaming checkpoints record the full history of the Kafka producer.