Skip to content

Instantly share code, notes, and snippets.

@aidin-foroughi
Last active September 29, 2023 00:23
Show Gist options
  • Select an option

  • Save aidin-foroughi/2ff62c4e19e620a1d6e2a512644fc89b to your computer and use it in GitHub Desktop.

Select an option

Save aidin-foroughi/2ff62c4e19e620a1d6e2a512644fc89b to your computer and use it in GitHub Desktop.
airflow

Put airflow learnings here

Essentially, you can use it to distribute work on a celery cluster very easily.

You typically want to use of the standard operators BashOperator PythonOperator or the best I think: DockerOperator

Note that communication between dag components happens over Variables or XCOM.

To pass data to the DockerOperator, you read from XCOM or Variables, and set some environment variables for the docker. For reading results back, by default, you get back the last printed line in to the stdout of the docker container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment