Scheduler
- The scheduler is responsible for monitoring tasks and DAGs (Directed Acyclic Graphs), triggering scheduled workflows, and distributing tasks to the executor.
- It parses DAG files at regular intervals and determines when tasks should be executed.
Executor
- The executor determines how and where tasks are executed.
- Different types of executors include:
  - SequentialExecutor: Runs one task at a time, useful for debugging.
  - LocalExecutor: Runs multiple tasks in parallel within the same machine.
  - CeleryExecutor: Distributes tasks across multiple worker nodes using Celery.
  - KubernetesExecutor: Runs tasks in separate Kubernetes pods.
Web Server (Airflow UI)
- Provides a web-based interface to manage and monitor workflows.
- Allows users to view DAGs, inspect task logs, trigger jobs manually, and manage configurations.
Metadata Database
- Stores DAG definitions, task states, execution logs, and other metadata.
- Typically uses PostgreSQL or MySQL in production.
- The scheduler and webserver interact with the metadata database to track workflow execution.
Workers (if using CeleryExecutor or KubernetesExecutor)
- Workers execute tasks assigned by the executor.
- In CeleryExecutor, multiple worker nodes process tasks in parallel.
- In KubernetesExecutor, each task runs in a separate Kubernetes pod.
DAGs (Directed Acyclic Graphs)
- DAGs define the workflows in Airflow.
- A DAG consists of tasks and their dependencies, written in Python.
Task Instances
- A task instance represents a specific run of a task within a DAG at a particular execution date.
- Task instances have different states such as running, success, failed, queued, etc.
Triggerer (for Deferrable Operators)
- Introduced in Airflow 2.x, the triggerer is responsible for handling deferred tasks asynchronously.
- Helps optimize resource usage by freeing up worker slots while waiting for external events.

Airflow Features

Apache Airflow provides following features:

Pure Python
Useful UI
Robust Integrations
Easy to Use
Open Source

Airflow Feature: Pure Python

No more command-line or XML black-magic! Everything is Python:

create workflows
extend
python libraries
scheduler, executor and workers run Python

Airflow Feature: Useful UI - DAG Filtering

all dags
active dags
paused dags
running dags
filter by tag
filter by name

Airflow Feature: Useful UI - Cluster Activity

Worker Status
Task Distribution
Queue Health
Executor Performance
Debuging

Airflow Feature: Robust Integrations

Airflow offers robust integrations with :

Cloud Platforms
Databases
BigData Frameworks

Airflow Feature: Easy to Use

Anyone with Python knowledge can deploy a workflow. Apache Airflow does not limit the scope of your pipelines; you can use it to build ML models, transfer data, manage your infrastructure, and more.

Airflow Feature: Open Source

Last commit: 1 hour ago
Total commits: +28k

Wherever you want to share your improvement you can do this by opening a PR. It’s simple as that, no barriers, no prolonged procedures. Airflow has many active users who willingly share their experiences. Have any questions? Check out our buzzing slack.

Airflow Providers packages

Airflow has 80+ providers packages includng integrations with third party integrations. They are updated independently of the Apache Airflow core. The current integrations are shown below

Airflow Docker stack

Airflow has an official Dockerfile and Docker image published in DockerHub as a convenience package for installation. You can extend and customize the image according to your requirements and use it in your own deployments.

Further Sources

Refer official documents on Apache Airflow here:

Airflow Documentation: https://airflow.apache.org/docs/
Airflow Usecases: https://airflow.apache.org/use-cases/

Table of Contents

What is Apache Airflow?

Airflow Principles

Airflow Principle: Scalable

Scaling Strategies

Airflow Principle: Dynamic

Airflow Principle: Extensible

Airflow Principle: Elegant

Airflow: Core Components