Airflow

New Features

AIR-190: You can now run Airflow on anaconda virtualenv. Users have the option of choosing the Python version while creating an Airflow cluster. Selecting Python 3.5 triggers the Airflow setup on the anaconda environment along with support for package management (ability to install/uninstall packages on the cluster from the Qubole UI) for the cluster. Beta, Via Support, Disabled
EAM-1334: Apache Airflow recommends flavors of MySQL and Postgres as database backends to store certain metadata that is required to run workflows. QDS has now added support for the AWS Aurora-MySQL RDS to be used as a database backend for Airflow content.
EAM-1271/AIR-195: For continuous development, integration, and deployment of Airflow DAGs, QDS has introduced tooling to upload Airflow Python DAG files to Amazon S3, edit them in-place, and sync them with Airflow clusters periodically (in the background). You can also download other Airflow relevant files such as process and log files from the Airflow cluster page. This aids quick, iterative development of complex workflows and keeps the files in sync. Beta, Via Support, Disabled, Cluster Restart Required. Learn more
AIR-163: The Monit httpd dashboard is now accessible through the UI for Airflow clusters. You can view the status and resource utilization for various process running within the cluster. The cluster administrator can start/stop/restart these processes using the UI. Cluster Restart Required

AIR-232: To prevent multiple Airflow cluster instances from overwriting to the same remote location for process_logs, QDS has changed the process_logs location to include the cluster instance ID. Cluster Restart Required. Learn more

AIR-234: The Qubole Authenticator’s change to reload user permissions on each request from the headers was returning to the old session-cookie based flow (whose implementation was removed) because of old clusters writing to the same cookie. To fix this, QDS has changed the SESSION_COOKIE_NAME setting on the Airflow Web server and now uses different cookie names based on the cluster ID. Cluster Restart Required
AIR-65: Added support in QuboleOperator to pass the use_customer_cluster, customer_cluster_label, and additional_options parameters while submitting DBImport and DBExport commands.