Airflow Xcom Exclusive [Full Version]
The TaskFlow API makes exclusive XComs cleaner by hiding the push/pull logic, but the discipline remains:
Example:
trigger = TriggerDagRunOperator( task_id='trigger_other', trigger_dag_id='consumer_dag', conf="xcom_value": " ti.xcom_pull(task_ids='producer_task') " ) airflow xcom exclusive
Master Airflow XCom: From Basics to Advanced Custom Backends The TaskFlow API makes exclusive XComs cleaner by
If exclusive XCom access is critical for correctness, consider: airflow xcom exclusive
: If you must handle larger data, you can set up a custom XCom Backend to store results in object storage like AWS S3 or GCS.
7 Comments
Pingback: What is your plan for 2017? - IAPAM : IAPAM
Pingback: Monday Morning Moment – Extreme Ownership – the Greater Good of Both Taking Responsibility and Letting Go of Control | Blog – Deb Mills
Pingback: Extreme Ownership: How U.S. Navy Seals Lead and Win – Michael Hammer