Create and grant permissions to service principal
If your selected the access method requires a service principal with adequate permissions, and you do not have one, follow these steps:
1. Create an Azure AD application and service principal that can access resources. Note the following properties:
* client-id: An ID that uniquely identifies the application.
* directory-id: An ID that uniquely identifies the Azure AD instance.
* service-credential: A string that the application uses to prove its identity.
2. Register the service principal, granting the correct role assignment, such as Storage Blob Data
3. Contributor, on the Azure Data Lake Storage Gen2 account.
Reference:
https://docs.databricks.com/data/data-sources/azure/azure-datalake-gen2.html

Reference:
https://azure.microsoft.com/en-us/blog/incrementally-copy-new-files-by-lastmodifieddate-with-azure-data-factory/
https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-tumbling-window-trigger

Explanation:
Scenario: Stage inventory data in Azure Data Lake Storage Gen2 before loading the data into the analytical data store. Litware wants to remove transient data from Data Lake Storage once the data is no longer in use. Files that have a modified date that is older than 14 days must be removed.
Service: Azure Data Factory
Clean up files by built-in delete activity in Azure Data Factory (ADF).
ADF built-in delete activity, which can be part of your ETL workflow to deletes undesired files without writing code. You can use ADF to delete folder or files from Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, File System, FTP Server, sFTP Server, and Amazon S3.
You can delete expired files only rather than deleting all the files in one folder. For example, you may want to only delete the files which were last modified more than 13 days ago.
Feature: Delete Activity
Reference:
https://azure.microsoft.com/sv-se/blog/clean-up-files-by-built-in-delete-activity-in-azure-data-factory/

Explanation
Create a function, with the help of a blob trigger template, which is triggered when files are uploaded to or updated in Azure Blob storage.
You use a consumption plan, which is a hosting plan that defines how resources are allocated to your function app. In the default Consumption Plan, resources are added dynamically as required by your functions. In this serverless hosting, you only pay for the time your functions run. When you run in an App Service plan, you must manage the scaling of your function app.
References:
https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-storage-blob-triggered-function

Scenario: You must maximize the performance of the Real Time Response system.
Design data processing solutions
Testlet 2
Case study
This is a case study. Case studies are not timed separately. You can use as much exam time as you would like to complete each case. However, there may be additional case studies and sections on this exam. You must manage your time to ensure that you are able to complete all questions included on this exam in the time provided.
To answer the questions included in a case study, you will need to reference information that is provided in the case study. Case studies might contain exhibits and other resources that provide more information about the scenario that is described in the case study. Each question is independent of the other questions in this case study.
At the end of this case study, a review screen will appear. This screen allows you to review your answers and to make changes before you move to the next section of the exam. After you begin a new section, you cannot return to this section.
To start the case study
To display the first question in this case study, click the Next button. Use the buttons in the left pane to explore the content of the case study before you answer the questions. Clicking these buttons displays information such as business requirements, existing environment, and problem statements. If the case study has an All Information tab, note that the information displayed is identical to the information displayed on the subsequent tabs. When you are ready to answer a question, click the Question button to return to the question.
Overview
You develop data engineering solutions for Graphics Design Institute, a global media company with offices in New York City, Manchester, Singapore, and Melbourne.
The New York office hosts SQL Server databases that stores massive amounts of customer data. The company also stores millions of images on a physical server located in the New York office. More than 2 TB of image data is added each day. The images are transferred from customer devices to the server in New York.
Many images have been placed on this server in an unorganized manner, making it difficult for editors to search images. Images should automatically have object and color tags generated. The tags must be stored in a document database, and be queried by SQL You are hired to design a solution that can store, transform, and visualize customer data.
Requirements
Business
The company identifies the following business requirements:
* You must transfer all images and customer data to cloud storage and remove on-premises servers.
* You must develop an analytical processing solution for transforming customer data.
* You must develop an image object and color tagging solution.
* Capital expenditures must be minimized.
* Cloud resource costs must be minimized.
Technical
The solution has the following technical requirements:
* Tagging data must be uploaded to the cloud from the New York office location.
* Tagging data must be replicated to regions that are geographically close to company office locations.
* Image data must be stored in a single data store at minimum cost.
* Customer data must be analyzed using managed Spark clusters.
* Power BI must be used to visualize transformed customer data.
* All data must be backed up in case disaster recovery is required.
Security and optimization
All cloud data must be encrypted at rest and in transit. The solution must support:
* parallel processing of customer data
* hyper-scale storage of images
* global region data replication of processed image data
Design data processing solutions
Testlet 3
Case study
This is a case study. Case studies are not timed separately. You can use as much exam time as you would like to complete each case. However, there may be additional case studies and sections on this exam. You must manage your time to ensure that you are able to complete all questions included on this exam in the time provided.
To answer the questions included in a case study, you will need to reference information that is provided in the case study. Case studies might contain exhibits and other resources that provide more information about the scenario that is described in the case study. Each question is independent of the other questions in this case study.
At the end of this case study, a review screen will appear. This screen allows you to review your answers and to make changes before you move to the next section of the exam. After you begin a new section, you cannot return to this section.
To start the case study
To display the first question in this case study, click the Next button. Use the buttons in the left pane to explore the content of the case study before you answer the questions. Clicking these buttons displays information such as business requirements, existing environment, and problem statements. If the case study has an All Information tab, note that the information displayed is identical to the information displayed on the subsequent tabs. When you are ready to answer a question, click the Question button to return to the question.
Background
Current environment
The company has the following virtual machines (VMs):

Requirements
Storage and processing
You must be able to use a file system view of data stored in a blob.
You must build an architecture that will allow Contoso to use the DB FS filesystem layer over a blob store. The architecture will need to support data files, libraries, and images. Additionally, it must provide a web-based interface to documents that contain runnable command, visualizations, and narrative text such as a notebook.
CONT_SQL3 requires an initial scale of 35000 IOPS.
CONT_SQL1 and CONT_SQL2 must use the vCore model and should include replicas. The solution must support 8000 IOPS.
The storage should be configured to optimized storage for database OLTP workloads.
Migration
* You must be able to independently scale compute and storage resources.
* You must migrate all SQL Server workloads to Azure. You must identify related machines in the on- premises environment, get disk size data usage information.
* Data from SQL Server must include zone redundant storage.
* You need to ensure that app components can reside on-premises while interacting with components that run in the Azure public cloud.
* SAP data must remain on-premises.
* The Azure Site Recovery (ASR) results should contain per-machine data.
Business requirements
* You must design a regional disaster recovery topology.
* The database backups have regulatory purposes and must be retained for seven years.
* CONT_SQL1 stores customers sales data that requires ETL operations for data analysis. A solution is required that reads data from SQL, performs ETL, and outputs to Power BI. The solution should use managed clusters to minimize costs. To optimize logistics, Contoso needs to analyze customer sales data to see if certain products are tied to specific times in the year.
* The analytics solution for customer sales data must be available during a regional outage.
Security and auditing
* Contoso requires all corporate computers to enable Windows Firewall.
* Azure servers should be able to ping other Contoso Azure servers.
* Employee PII must be encrypted in memory, in motion, and at rest. Any data encrypted by SQL Server must support equality searches, grouping, indexing, and joining on the encrypted data.
* Keys must be secured by using hardware security modules (HSMs).
* CONT_SQL3 must not communicate over the default ports
Cost
* All solutions must minimize cost and resources.
* The organization does not want any unexpected charges.
* The data engineers must set the SQL Data Warehouse compute resources to consume 300 DWUs.
* CONT_SQL2 is not fully utilized during non-peak hours. You must minimize resource costs for during non- peak hours.