How to install Apache Airflow with Poetry

Apache Airflow was created by Maxime Beauchemin at Airbnb and is a tool used and loved by many data engineers all around the world. Airflow is used to manage and orchestrate the data pipelines you may have in order to fulfill your many data requirements. Without Airflow, you'd have to engineer your own logging and monitoring system, your own scheduling system, and triggering mechanisms.

Airflow was open-sourced as a part of the Apache Software Foundation in 2016 but despite its maturity, manually installing it using more modern dependency managers is a pain. There is no official support for Poetry or Pipenv despite the fact that having a REAL dependency manager is a must have for anyone using Python. Yes, that includes you data scientists and analysts that think you can get away with raw dogging venv and pip. You can't get away with it.

"He Cant Keep Getting Away With It"-meme

Good package management is a must have if you want to make your applications installable. Developing it once it surely not what you want. You probably want to share your code with other people or deploy the server somewhere else other than running it on you local machine. Poetry and Pipenv ensure deterministic reproducible applications by generating a .lock file and updating each time you install new packages. Now, everyone can install the project and run your code without hitting missing dependency problems.

And as such, I created this guide to help fellow Poetry enthusiasts with installing Airflow on their machines.

Copy and paste the following code into your project's pyproject.toml-file:

# pyproject.toml

[tool.poetry]
authors = ["qwertyu_alex"]
description = ""
name = ""
readme = "README.md"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.group.airflow.dependencies]
apache-airflow = {version = "2.7.3", python = ">=3.10,<3.12"}
apache-airflow-providers-common-sql = "1.8.0"
apache-airflow-providers-ftp = "3.6.0"
apache-airflow-providers-http = "4.6.0"
apache-airflow-providers-imap = "3.4.0"
apache-airflow-providers-slack = "^8.5.1"
apache-airflow-providers-sqlite = "3.5.0"
Flask = "2.2.5"
Flask-AppBuilder = "4.3.6"
Flask-Babel = "2.0.0"
Flask-Caching = "2.1.0"
Flask-JWT-Extended = "4.5.3"
Flask-Limiter = "3.5.0"
Flask-Login = "0.6.3"
Flask-SQLAlchemy = "2.5.1"
Flask-Session = "0.5.0"
Flask-WTF = "1.2.1"
Jinja2 = "3.1.2"
grpcio = "1.59.2"
gunicorn = "21.2.0"
oauth2client = "^4.1.3"

[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core"]

Then create a lock file for your Poetry project:

poetry lock

And finally install Airflow:

poetry install