Raw data is rarely analysis-ready. In this project, I build a complete pipeline that takes a dataset from Kaggle, lands it in Azure Data Lake Storage (ADLS), cleans and prepares it in Databricks, and delivers refined data to a “gold” container for analysis — with Azure Data Factory (ADF) orchestrating every step and Databricks running the Python transformation code.

By the end, you’ll understand how the pieces fit together: the architecture, the setup, the orchestration, and how to monitor the pipeline once it runs.

What the pipeline does

  • Extracts data from a Kaggle dataset
  • Loads it into an ADLS “bronze” container (raw data)
  • Transforms it in Databricks — cleaning and preparation with Python and Spark
  • Loads the refined data into an ADLS “gold” container (analysis-ready)

Prerequisites

  • An Azure subscription
  • Basic knowledge of Azure services, particularly Azure Data Factory and Azure Databricks
  • A Kaggle account to access datasets
  • Python programming skills for writing the transformation scripts

Architecture overview

  • Data extraction: Download the Kaggle dataset
  • Data ingestion: Upload the dataset to the ADLS bronze container
  • Data transformation: Clean and prepare the data in Databricks
  • Data loading: Write the transformed data to the ADLS gold container

Step-by-step implementation

1. Set up the Azure resources

  • Azure Data Lake Storage: Create a storage account with two containers — bronze for raw, unprocessed data and gold for cleaned, processed data. (In the Azure portal, search for “Data Lake Storage” and create the account, then add both containers.)
  • Azure Data Factory: Search for “Data Factory” in the portal, create an instance, and launch ADF Studio.
  • Azure Databricks: Search for “Databricks” in the portal, create a workspace, launch it, and create a cluster.

2. Get the Kaggle dataset into bronze

  • Log in to Kaggle, navigate to the dataset you want, and download it.
  • Upload it to the bronze ADLS container using the Azure portal or Azure Storage Explorer.

3. Configure Azure Data Factory

Create the two connections the pipeline needs:

  • Linked service for ADLS — in ADF, go to the Manage tab and create a linked service for your ADLS account.
  • Linked service for Databricks — create a second linked service connecting ADF to your Databricks workspace.

4. Write the Databricks cleaning notebook

In your Databricks workspace, create a notebook that reads from bronze, cleans the data, and writes to gold. The cleaning step is where the real work happens — removing duplicates, handling missing values, transforming data types, and filtering out unwanted data:

# Read data from the ADLS bronze container
df = spark.read.format("csv").option("header", "true").load(
    "abfss://bronze@youradlsaccount.dfs.core.windows.net/your_dataset.csv"
)

# Data cleaning and preparation
df_clean = df.dropDuplicates() \
             .na.fill("Unknown") \
             .filter("age > 18") \
             .withColumn("salary", df["salary"].cast("double"))

# Write cleaned data to the ADLS gold container
df_clean.write.mode("overwrite").format("parquet").save(
    "abfss://gold@youradlsaccount.dfs.core.windows.net/cleaned_dataset.parquet"
)

Make sure the notebook is configured to run on the cluster you created.

5. Build the pipeline in ADF

  • Create a new pipeline.
  • Add a Copy activity: source = the bronze container, sink = the Databricks file system (DBFS).
  • Add a Databricks Notebook activity, pointed at the notebook from Step 4.
  • Add a second Copy activity: source = DBFS, sink = the gold container.
  • Add triggers so the pipeline runs on a schedule or in response to events.

6. Orchestrate and monitor

Azure Data Factory gives you the tools to run this reliably:

  • Pipeline triggers — schedule runs at specific times or in response to events
  • Monitoring dashboard — watch the status of every activity, view logs, and troubleshoot issues
  • Alerts and notifications — get notified about pipeline failures or other important events

Key takeaways

  • Azure Data Factory is a powerful tool for orchestrating complex data workflows.
  • Azure Databricks provides a robust platform for data cleaning and transformation using Python.
  • Azure Data Lake Storage offers scalable, secure storage for both raw and processed data.
  • Monitoring is not optional — it’s how you know the pipeline is healthy and how you fix it fast when it isn’t.

Built this way, the pipeline is efficient, flexible, and scalable — ready to adapt as data requirements and business needs change.

Next steps: New to how the data roles fit together? Read Understanding Data Engineering and How It Differs from Data Analysis, Data Science, and ML Engineering. Want the concepts behind this build? Browse the latest articles or get in touch.

Scroll to Top