Real world project for Azure Data Engineers using Azure Data Factory, SQL, Data Lake, Databricks, HDInsight and PowerBI

Leave a Comment / Uncategorized / By
Links to Read:
  1. Introduction to Azure Storage
  2. What is Azure SQL Database
  3. What is Azure Synapse Analytics
  4. What is Azure Cosmos DB
  5. What is Azure Data Lake Storage Gen 2
  6. Copy Activity (ADF)
  7. Linked Services (ADF)
  8. Datasets (ADF)
  9. Pipelines (ADF)
  10. Validation Activity (ADF)
  11. Get Metadeta (ADF)
  12. If Condition (ADF)
  13. Web Activity (ADF)
  14. Delete Activity (ADF)
  15. Quickstart – Create data factory (ADF)
  16. Create Storage Account (ADF)
  17. Create Storage Account with Azure Data Lake Storage Gen2 (ADF)
  18. Quickstart – Create database (ADF)

ECDC Datasets: 

  1. Data on 14-day notification rate of new COVID-19 cases and deaths
  2. Data on testing for COVID-19 by week and country
  3. Data on country response measures to COVID-19
  4. Data on hospital and ICU admission rates and current occupancy for COVID-19

Github: 

  1. https://github.com/cloudboxacademy/covid19

Common Questions

1. Explain Azure Data Lake Gen2 vs Blob Storage

Azure Data Lake Gen 2

Azure Blob Storage

Actual hierarchy folder structure
Fine grained access controls (ACL)
No soft delete (yet)
Cost more than blob (2-3x)

–> For Analytics / Data Warehousing

“Logical” hierarchy folder structure
Access control at resource level
Soft delete
Cheaper

–> General purpose storage

1. ADF Parameters vs Variables

Parameters

Variables

External values passed into pipelines, datasets or linked services. Values cannot be changed

Internal values set inside a pipeline. Value can be changed inside the pipeline using Set Variable or Append Variable Activity

Leave a Comment

Your email address will not be published. Required fields are marked *