File and Partition Sensors

Sensors are a certain type of operator that keep running until a certain criterion is met. Qubole supports:

• File sensor - It monitors a file’s availability at its location.

• Hive partition sensor - It monitors the partition’s availability on its parent Hive table. It also monitors the different columns in the Hive table.

Currently, Qubole provides only API support to create file and partition sensors. For more information, see sensor-api-index.

Airflow uses file and partition sensors for programmatically monitoring workflows.