When you enroll through our links, we may earn a small commission—at no extra cost to you. This helps keep our platform free and inspires us to add more value.
End to End Azure Data Engineering Real Time Project- 3 Hours
Azure Data Engineer Project using the modern Data Tools like ADF, Azure Databricks, Azure Synapse Analytics, ADLS Gen2
This Course Includes
udemy
4.6 (397 reviews )
3 total hours 
english
Online - Self Paced
course
Udemy
About End to End Azure Data Engineering Real Time Project- 3 Hours
In this course, let's build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
The tools that are covered in this project are,
Azure Data Factory
Azure Data Lake Storage Gen2
Azure Databricks
Azure Synapse Analytics
Azure Key vault
Microsoft Entra ID (Previously called as AAD) and
Microsoft Power BI
The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Microsoft Entra ID ( Previously called as AAD) and Azure Key Vault for the monitoring and governance purpose. In this video, I have also covered the complete end to end pipeline testing right from how a new data gets ingested followed by the data transformation until it goes updating the report that we will be creating using the Power BI
What You Will Learn?
- Learn how to efficiently ingest data from an on-premise SQL Server database to Azure Data Lake Storage Gen2 using Azure Data Factory..
- Leverage Azure Data Lake Storage for Scalable Data Storage- Understand how to utilize Azure Data Lake Storage Gen2 for scalable and secure data storage..
- Dive into data transformation using Azure Databricks. Learn how to clean, process, and enrich raw data, converting it into the most refined form for analytics..
- Load and Query Data in Azure Synapse Analytics- Discover how to load transformed data into Azure Synapse Analytics and perform powerful queries..
- Build Dashboards with Microsoft Power BI- Get hands-on experience with Microsoft Power BI to create interactive dashboards that provide real-time insights..
- Implement Security with Microsoft Entra ID (Previously AAD). Understand the role of Microsoft Entra ID in securing your data platform.
- Explore how Azure Key Vault helps in securely storing and managing sensitive information like secrets, keys, and certificates..
- Test and Monitor End-to-End Data Pipelines- Gain skills in testing and monitoring your entire data pipeline, from data ingestion to reporting..