Where does parallelism appear in Airflow?

Parallelism in Apache Airflow manifests at several levels:

1. Task-level parallelism within a DAG — Airflow allows running multiple tasks simultaneously if they do not depend on each other. This is achieved through the use of worker pools and parallelism settings.

2. DAG-level parallelism — multiple DAGs can be run concurrently, enabling processing of different data streams in parallel.

3. Worker-level parallelism — Airflow uses a distributed architecture with multiple workers that can execute tasks in parallel.

4. Parallelism within a single task — if a task is implemented with multithreading or multiprocessing support (e.g., in PythonOperator), parallelism can be realized within the task itself.

Settings affecting parallelism:

• parallelism — a global parameter limiting the total number of tasks that can run simultaneously.
• dag_concurrency — the maximum number of tasks that can run concurrently within a single DAG.
• max_active_runs_per_dag — the maximum number of concurrent runs of a single DAG.
• pool — allows limiting parallelism for groups of tasks.

Example of parallelism configuration in airflow.cfg:

[core]
parallelism = 32

[scheduler]
dag_concurrency = 16
max_active_runs_per_dag = 4

Thus, parallelism in Airflow enables efficient resource utilization and accelerates the execution of complex workflows.