Kuidas edastada suuri andmeid Airflow's ülesannete vahel?

In Airflow, it is not recommended to use XCom for transferring large data between tasks, as it is intended for small messages (usually up to 48 KB). Instead, it is better to use external storage:

• Store data in cloud storage (S3, GCS, Azure Blob) or databases.
• Pass links (paths, keys) to these data between tasks via XCom.

This approach allows efficient handling of large data volumes and avoids overloading Airflow's metadata.

Example:

# In the first task, save data to S3 and pass the key
s3_key = upload_to_s3(data)
ti.xcom_push(key='s3_key', value=s3_key)

# In the second task, retrieve the key and load data
s3_key = ti.xcom_pull(key='s3_key')
data = download_from_s3(s3_key)