#Azure Data Factory

What’s Azure Data Factory?

  • A fully managed, serverless data integration solution for ingesting, preparing and transforming all the data at scale.
  • Problem Statement:
  • Businesses are generating more and more data everyday. This data is coming from a variety of sources such as Multi-Cloud applications, On Premises applications and SaaS applications majorly.
  • The rate of this data growth is also exponential.
  • the format of this data is also significantly varied – Structured, Semi – Structured and Unstructured.
  • Microsoft Data Factory:
  • ADF helps business ingest data for all the above sources and formats, analyze and transform data, and finally publishing the dashboards.
  • ADF can also be used to orchestrate workflow from start to finish. It offer monitoring capability using which you can monitor each data pipeline.

What does it mean “ADF is a fully managed service”?

Once an ADF is created in your service, Microsoft will take care of all the management in Data Factory. We don’t have to know or manage any technical aspects behind this service. Also Microsoft takes care of the scalability, availability and security aspects of the ADF service.

What does it mean ADF has a Serverless Infrastructure?

This means the compute environment can be scaled to any level without having to worry about infrastructure management.

Azure Data factory is Not a:

  • Data Migration Tool to be used while transferring data from one database to another.
  • Data Streaming Services.
  • Complex data transformations.
  • Data storage solution.

Storage Solutions

Key factors to consider:

  1. Structure of the data
    • Structured: Rows & Columns in Spread Sheet
    • Semi-Structured: JSON, XML
    • Un-Structured: System logs, Photos, videos
  2. Operational Needs
    • How Often the data is accessed
    • How quickly do we need to serve
    • Need to run simple queries
    • Need to run heavy analytical workload
    • Accessed from multiple regions

Azure Databases:

  • Azure SQL Databases – Fully managed cloud based DB
  • Azure Database for MySQL – FM – MySQL DB as a Service
  • Azure Database for PostgreSQL
  • Azure Database for MariaDB
  • VM Images with Oracle, SQL Server

Azure Storage Accounts:

  • Blob Storage – can store unstructured data
  • File Storage – FM could file store which can be accessed from anywhere by SNP protocol
  • Disk Storage – VM disks
  • Table Storage – Structured data as Key Value pairs
  • Queue Storage – Store and retrieve message from queues.

Azure Storage Solutions:

  • Azure Data lake Storage Gen 2 – Built on top of blob storage – to service petabytes of data.
  • Azure Cosmos DB – Fully managed NoSQL database