Upgrading Airflow Clusters

Upgrading Airflow Cluster Version on QDS

Airflow 1.8.2 is supported with MySQL 5.6 or later versions and Airflow 1.10.0 or above versions require MySQL 5.6.4 or later versions. If you are not using MySQL on the cluster, ensure that you are using a compatible version before you proceed to upgrade.

Follow these instructions to upgrade the Airflow cluster version:

  1. Log into Airflow cluster.

  2. Execute the following commands to stop Airflow scheduler and workers.

    sudo monit stop scheduler; sudo monit stop worker;

  3. Keep a backup of all the DAGs that are present in $AIRFLOW_HOME/dags directory.

  4. Check if all workers have completed running tasks that were being executed. Run ps -ef|grep airflow and wait until there is no airflow run commands running. This is to ensure that there is no task left in an inconsistent state.

  5. Navigate to the Clusters page.

  6. Clone the existing cluster and update the cloned clusters version to the desired version.

  7. Log into the cloned cluster.

  8. Restore the backed up DAGs under $AIRFLOW_HOME/dags. You have successfully upgraded the Airflow cluster version on QDS.

Note

While upgrading the version of Airflow clusters, it is recommended to upgrade intermediate versions and one version of Airflow cluster at a time. For example, if you want to upgrade from version 1.8.2 to 1.10.2, you should first upgrade it to 1.10.0 and finally to 1.10.2. If you try to upgrade it directly from 1.8.2 to 1.10.2, it may cause issues while running database migration and result in an inconsistent state of the database.

Upgrading Long-running Airflow Clusters

If your airflow cluster has been running for a longer time with multiple DAGs, some tables such as task_instance will contain large data. This causes a severe delay in database migration (much more than the allowed cluster start time). As a result, cluster startup is aborted abruptly causing the database to go into an inconsistent state. To avoid this, you must run the database migration manually before you upgrade.

Follow these instructions to upgrade long-running Airflow clusters:

  1. Copy the values of the following settings from your old cluster.

    • core.sql_alchemy_conn

    • celery.result_backend

  2. Take a backup of your DAGs and shut down the old cluster.

  3. Start a new Airflow cluster with the newer version of Airflow and default cluster Datastore.

    Important

    Do not select any external Datastore.

    This is only a temporary cluster to run the migrations. Ensure that the fernet key is the same as the old cluster.

  4. Modify the values of the copied settings (refer to Step 1) in the new cluster’s /usr/lib/airflow/airflow.cfg file.

  5. Run the following commands on the temporary cluster to execute the migration. It is recommended to use tmux to avoid session timeout issues or use Analyze to run a Shell command on this new cluster.

    source /usr/lib/airflow/airflow/qubole_assembly/scripts/virtualenv.sh activate
airflow upgradedb
source /usr/lib/airflow/airflow/qubole_assembly/scripts/virtualenv.sh deactivate

    This might take some time based on the size of the database.

  6. After completion, shut down the temporary cluster.

  7. Clone the old Airflow cluster to create one with a newer version of Airflow. Change the airflow version while cloning.

  8. Start the new cluster to get things up and running on the new Airflow version. You have successfully upgraded a long-running Airflow cluster.