Airflow

New Features

  • AIR-523: Since the Jupyter Notebook Command is now available in QuboleOperator as jupytercmd, the users can schedule their Jupyter Notebooks in Airflow (Cluster Restart Required). For more information, see Qubole Operator Examples.

  • AIR-501: This feature provides the ability to pass the argument arguments=['true'] in Qubole Operator’s get_result() method to retrieve headers (Cluster Restart Required).

  • AIR-500: Now, Qubole supports Apache Airflow version 1.10.9QDS (beta). This Airflow version is supported only with python version 3.7. For more information, see QDS Components: Supported Versions and Cloud Platforms.

    Since the last Qubole supported version was 1.10.2QDS, there are a lot of new additions in features and improvements such as DynamoDB to S3 operator, BigQuery to Mysql Operator, newly added GcsToGDriveOperator, support for optional configurations while triggering DAGs, configurable tags on DAGs, persisting serialised DAGs for webserver scalability, and so on. For more information on the new features and improvements, see Apache Airflow Changelog.

  • AIR-272: Now, the users have an option on the Clusters page under the cluster details to delete the DAGs uploaded by the user through DAG explorer (Cluster Restart Required). For more information, see Deleting a DAG on an Airflow Cluster.

Bug Fixes

  • AIR-407: Airflow clusters must be provided with a valid fernet key (a 32 bytes long key). This improvement helps to prevent the error that used to appear while adding the key-value pairs for new variables.