Did you write Airflow DAGs and jobs manually or using templates?

When writing Airflow DAGs and jobs, I try to avoid manual copying and pasting of code ("copy-pasting by hand"). Instead, I use templates and code generation to increase reusability and reduce the likelihood of errors.

For example, if I have several similar DAGs with different parameters, I can create a function that takes parameters and returns a DAG object. This allows me to quickly create new DAGs without duplicating code:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def create_dag(dag_id, schedule, default_args):
    def task_function(**kwargs):
        print(f"Running task in {dag_id}")

    dag = DAG(dag_id, schedule_interval=schedule, default_args=default_args)

    with dag:
        task = PythonOperator(task_id='print_task', python_callable=task_function)

    return dag

# Example of creating multiple DAGs
for env in ['dev', 'prod']:
    dag_id = f'my_dag_{env}'
    globals()[dag_id] = create_dag(dag_id, '@daily', {'start_date': datetime(2023, 1, 1)})

This approach simplifies maintenance and scaling of pipelines.