What is Apache Airflow?

Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows in the form of directed acyclic graphs (DAGs). It is used for managing ETL/ELT processes, scheduling tasks, and automating various operations.

Main components:

• Scheduler: Responsible for triggering tasks and dispatching them for execution.
• Webserver: Provides a user interface for monitoring DAGs, managing tasks, and viewing logs.
• Worker: Executes tasks (Operators).
• Database: Stores Airflow state, DAG metadata, task information, and runs.

Features and advantages:

• Python: Workflows are written in Python.
• Dynamic: DAGs can be generated dynamically.
• Extensible: Easily create custom operators and hooks.
• Scalable: Architecture allows horizontal scaling of components.
• GUI: Informative web interface for monitoring and debugging.

Example of a simple DAG:

from __future__ import annotations

import pendulum

from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator

with DAG(
    dag_id="simple_bash_dag",
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    schedule=None,
    catchup=False,
    tags=["example"],
) as dag:
    task1 = BashOperator(
        task_id="print_date",
        bash_command="date",
    )

    task2 = BashOperator(
        task_id="sleep_5",
        bash_command="sleep 5",
    )

    task3 = BashOperator(
        task_id="print_hello",
        bash_command="echo hello",
    )

    task1 >> task2 >> task3 # Defines task execution order