Given the information that the model is registered with MLflow and assuming predict is the method used to apply the model to a set of columns, we use the model.predict() function to apply the model to the DataFrame df using the specified columns. The model.predict() function is designed to take in a DataFrame and a list of column names as arguments, applying the trained model to these features to produce a predictions column.
When working with PySpark, this predictions column needs to be selected alongside the customer_id to create a new DataFrame with the schema customer_id LONG, predictions DOUBLE.
References:
* MLflow documentation on using Python function models:
https://www.mlflow.org/docs/latest/models.html#python-function-python
* PySpark MLlib documentation on model prediction:
https://spark.apache.org/docs/latest/ml-pipeline.html#pipeline

Explanation
The answer is AUTO LOADER,
Use Auto Loader instead of the COPY INTO SQL command when:
*You want to load data from a file location that contains files in the order of millions or higher. Auto Loader can discover files more efficiently than the COPY INTO SQL command and can split file processing into multiple batches.
*COPY INTO only directory listing but AUTO LOADER supports File notification method where the Auto Loader continues to ingest files as they arrive in cloud object storage lever-aging cloud provider(Queues and triggers) and Spark's structured streaming.
*Your data schema evolves frequently. Auto Loader provides better support for schema in-ference and evolution. See Configuring schema inference and evolution in Auto Loader.

Maintaining data quality rules in a centralized Delta table allows for the reuse of these rules across multiple DLT (Delta Live Tables) pipelines. By storing these rules outside the pipeline's target schema and referencing the schema name as a pipeline parameter, the team can apply the same set of data quality checks to different tables within the pipeline. This approach ensures consistency in data quality validations and reduces redundancy in code by not having to replicate the same rules in each DLT notebook or file.
:
Databricks Documentation on Delta Live Tables: Delta Live Tables Guide