1 min read

What is Airflow?

Airflow is a platform to programmatically author, schedule, and monitor workflows or data pipelines.
What is Airflow?

Airflow is a Workflow engine which means:

  • Manage scheduling and running jobs and data pipelines
  • Ensures jobs are ordered correctly based on dependencies
  • Manage the allocation of scarce resources
  • Provides mechanisms for tracking the state of jobs and recovering from failure.

It is highly versatile and can be used across many many domains:

https://airflow-tutorial.readthedocs.io/en/latest/_images/uses.png

Basic Airflow concepts

  • Task: a defined unit of work (these are called operators in Airflow)
  • Task instance: an individual run of a single task. Task instances also have an indicative state, which could be “running”, “success”, “failed”, “skipped”, “up for retry”, etc.
  • DAG: Directed acyclic graph, a set of tasks with explicit execution order, beginning, and end
  • DAG run: individual execution/run of a DAG

Read More...

Airflow basics — Airflow tutorial documentation
https://airflow-tutorial.readthedocs.io/en/latest/airflow-intro.html

#probyto #probytoai #datascience #machinelearning #python #artificialintelligence #ai #dataanalytics #data #bigdata #deeplearning #programming #datascientist #technology #coding #datavisualization #computerscience #pythonprogramming #analytics #tech #dataanalysis #iot #programmer #statistics #developer #ml #business #innovation #coder #dataanalyst

Subscribe and follow us for latest news in Data Science, Machine learning, technology and stay updated!

Facebook: https://facebook.com/probyto
Twitter: https://twitter.com/probyto
LinkedIn: https://linkedin.com/company/probyto
Instagram: https://instagram.com/probyto