Registering a DAG on an Airflow ClusterΒΆ

You can submit Airflow commands through a QDS shell command on an Airflow cluster. You can execute all Airflow commands available in the CLI using a shell command.

Note

Qubole supports file and Hive table sensors that Airflow can use to programmatically monitor workflows. For more information, see File and Partition Sensors and sensor-api-index.

Perform the following steps to register a DAG:

  1. Navigate to the Analyze page and click Compose. Select Shell Command from the Command Type drop-down list.

  2. Bash Commands is selected by default from the drop-down list.

  3. Select the Airflow cluster. In the Bash Commands text field, enter a command, for example:

    • For AWS:

      cd ${AIRFLOW_HOME}/dags; sudo s3cmd -c /usr/lib/hustler/s3cfg get s3://<DAG location on S3> --force
      
    • For Azure:

      sudo /usr/lib/hadoop2/bin/hadoop dfs -Dfs.azure.account.key.${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net=${AZURE_STORAGE_ACCESS_KEY} -copyToLocal -f <REMOTE_LOC> ${AIRFLOW_HOME}/dags
      

      REMOTE_LOC must be in the form wasb://<container_name>@<blob_name>.blob.core.windows.net/<relative_loc>

    • For Oracle:

      sudo /usr/lib/hadoop2/bin/hadoop dfs -Dfs.oraclebmc.client.auth.tenantId=<ORACLE_TENANT_ID> -Dfs.oraclebmc.client.auth.userId=<ORACLE_USER_ID>  -Dfs.oraclebmc.client.auth.fingerprint=<ORACLE_FINGERPRINT> -Dfs.oraclebmc.client.auth.pemfilecontent="<ORACLE_PEMFILE>" -copyToLocal  -f  oci://bucket@path  ${AIRFLOW_HOME}/dags
      

      You can also use the following way to run this command:

      sudo /usr/lib/hadoop2/bin/hadoop dfs -Dfs.oci.client.auth.tenantId="<compute_tenant_id>" -Dfs.oci.client.auth.userId="<compute_user_id>"  -Dfs.oci.client.auth.fingerprint="<compute_key_finger_print>" -Dfs.oci.client.auth.pemfilepath="/path/to/pemfile" -copyToLocal  -f oci://<bucket>@<path> ${AIRFLOW_HOME}/dags
      

      REMOTE_LOC must be in the form oci://<bucket>@<namespace>/path

  4. Click Run to execute the query. Click Save if you want to run the same query later. (See Repo for more information on saving queries.)

  5. The query result is displayed in the Results tab.

The following figure shows an example of registering an Airflow DAG on AWS and the successful result.

../../../_images/ComposeAirflow.png

For more information, see Questions about Airflow.