Upgrading Airflow Clusters¶
Upgrading Airflow Cluster Version on QDS¶
Airflow 1.8.2 is supported with MySQL 5.6 or greater versions only while Airflow 1.10.0+ versions require MySQL 5.7 or greater versions. If you are not using MySQL on the cluster, ensure before upgrading that you are using a compatible version.
Follow these instructions to upgrade the Airflow cluster version:
Log into Airflow cluster.
Execute the following commands to stop Airflow scheduler and workers.
sudo monit stop scheduler; sudo monit stop worker;
Keep a backup of all the DAGs that are present in
Check if all workers have completed running tasks that were being executed. Run
ps -ef|grep airflowand wait until there is no airflow run commands running. This is to ensure that there is no task left in an inconsistent state.
Navigate to the Clusters page.
Clone the existing cluster and update the cloned clusters version to the desired version.
Log into the cloned cluster.
Restore the backed up DAGs under
$AIRFLOW_HOME/dags. You have successfully upgraded the Airflow cluster version on QDS.
While upgrading the version of Airflow clusters, it is recommended to upgrade intermediate versions and one version of Airflow cluster at a time. For example, if you want to upgrade from version 1.8.2 to 1.10.2, you should first upgrade it to 1.10.0 and finally to 1.10.2. If you try to upgrade it directly from 1.8.2 to 1.10.2, it may cause issues while running database migration and result in an inconsistent state of the database.
Upgrading Long-running Airflow Clusters¶
If your airflow cluster has been running for a longer time with multiple DAGs, some tables such as
task_instance will contain
large data. This causes a severe delay in database migration (much more than the allowed cluster start time). As
a result, cluster startup is aborted abruptly causing the database to go into an inconsistent state. To avoid this, you
must run the database migration manually before you upgrade.
Follow these instructions to upgrade long-running Airflow clusters:
Copy the values of the following settings from your old cluster.
Take a backup of your DAGs and shut down the old cluster.
Start a new Airflow cluster with the newer version of Airflow. This is only a temporary cluster to run the migrations. Ensure that the fernet key is the same as the old cluster.
Modify the values of the copied settings (refer to Step 1) in the new cluster’s
Run the following commands on the temporary cluster to run the migration. It is recommended to use
tmuxto avoid session timeout issues.
For python 2.7 cluster:
For Python 3.5 cluster:
source $AIRFLOW_VIRTUALENV_LOC/bin/activate $AIRFLOW_VIRTUALENV_LOC
airflow initdband wait until the completion. This might take some time based on the size of the database.
After completion, shut down the temporary cluster.
Clone the old Airflow cluster to create one with a newer version of Airflow. Change the airflow version while cloning.
Start the new cluster to get things up and running on the new Airflow version. You have successfully upgraded a long-running Airflow cluster.