{"id":3454,"date":"2025-09-09T18:08:14","date_gmt":"2025-09-09T23:08:14","guid":{"rendered":"https:\/\/datacracy.co\/?p=3454"},"modified":"2026-05-27T11:58:00","modified_gmt":"2026-05-27T16:58:00","slug":"integrating-airflow-with-databricks","status":"publish","type":"post","link":"https:\/\/www.datacracy.co\/es\/integrating-airflow-with-databricks\/","title":{"rendered":"Integrating Airflow with Databricks"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

BLOG<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t
\n\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\n\t\t\t\tIntegrating Airflow with Databricks\t\t\t\t\t- Overview of Airflow, Databricks, and dbt\t\t\t\t \n\t\t\t<\/h2 > \n\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Before diving into the integration steps, let\u2019s quickly overview the technologies we\u2019ll be using.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t<\/path><\/svg>\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\u00a0**Apache Airflow**\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/h3>\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\tAn open-source platform used to programmatically author, schedule, and monitor workflows.\t\t\t\t\t<\/p>\n\t\t\t\t\n\t\t\t<\/div>\n\t\t\t\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t<\/path><\/svg>\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t**Databricks**\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/h3>\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\tA unified data analytics platform that provides a Spark-based processing engine, allowing you to build scalable data pipelines.\t\t\t\t\t<\/p>\n\t\t\t\t\n\t\t\t<\/div>\n\t\t\t\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t<\/path><\/svg>\t\t\t\t<\/span>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t**DBT (data build tool)**\t\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/h3>\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t

\n\t\t\t\t\t\tA tool that enables data analysts and engineers to transform data in a warehouse via SQL, with the added benefit of validating the data quality through testing and documentation.\t\t\t\t\t<\/p>\n\t\t\t\t\n\t\t\t<\/div>\n\t\t\t\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

By combining these tools, you can orchestrate a full end-to-end data pipeline in which data is processed in Databricks, and dbt is used to validate the quality of that data.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Setting Up Airflow<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Start by installing and setting up Apache Airflow. If you already have it running, you can skip this section.<\/p>

  • Install Airflow via pip<\/strong>: pip install apache-airflow.<\/li>
  • Initialize the database and start the webserver<\/strong>:
    airflow db init
    airflow webserver
    airflow scheduler
    Next, configure Airflow to work with Databricks.<\/li>
  • Airflow Configuration for Databricks<\/strong>:
    • Install the Databricks integration package: pip install apache-airflow-providers-databricks.<\/li>
    • Configure the Databricks connection in Airflow\u2019s UI:<\/li>
    • Go to the Airflow Admin tab \u2192 Connections \u2192 Add a new connection.<\/li>
    • Set **Conn Type** to Databricks.<\/li>
    • Add your **Databricks host URL** and **token**.<\/li><\/ul><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"channels4_profile\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t
      \n\t\t\t
      <\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
      \n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Running a Databricks Job from Airflow<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      To orchestrate a Databricks job in Airflow, you can use the DatabricksSubmitRunOperator. Here\u2019s a simple example of a
      DAG that triggers a Databricks job.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

      \n\t\t\t\t
      \n\t\t\t\t\t
      \n
      \n
      \n
      \n
      \n \ud83d\udc0d<\/span>\n <\/div>\n
      \n \n DatabricksJob.py <\/span>\n <\/div>\n <\/div>\n <\/div>\n\n
      \n
      \n