Airflow

New Features

  • AIR-290: QDS Azure now supports Airflow package management.

  • AIR-279: You can use your own Prometheus setup and get metrics from https://<env>.qubole.com/airflow-webserver-<cluster_id>/admin/metrics/ using the Prometheus Auth Token you have configured. Cluster Restart Required.

  • AIR-202: QDS now supports Apache Airflow v1.10.0. You can choose it froom the Airflow Version drop-down list on the Clusters page of the QDS UI. It includes new functionality including timezone support, performance optimization for large DAGs, and Kubernetes Operator and Executor. The complete changelog is here.

    Note

    • Airflow 1.10.0 also provides a web interface for Role-Based Access Control (RBAC), but this is not yet supported in QDS.

    • If you are using MySql or MariaDB as a database back end for your Airflow cluster, timezone support is not available because of limitations in those database systems.

    • You must create a new cluster to start using Airflow 1.10.0.

Bug Fixes

  • AIR-277: The default cluster data store for Airflow clusters has changed from MySQL to PostgreSQL. Cluster Restart Required.

  • AIR-266: postgres python driver is now pre-installed on Airflow clusters to facilitate connecting an Airflow cluster to a PostgreSQL data store. Cluster Restart Required.