8. Questions on Airflow Service Issues¶
Here is a list of FAQs that are related to Airflow service issues with corresponding solutions.
Which logs do I look up for Airflow cluster startup issues?
Refer to Airflow Services logs which are brought up during the cluster startup.
Where can I find Airflow Services logs?
Airflow services are Scheduler, Webserver, Celery, and RabbitMQ. The service logs are available at
/media/ephemeral0/logs/airflowlocation inside the cluster node. Since airflow is single node machine, logs are accessible on the same node. These logs are helpful in troubleshooting cluster bringup and scheduling issues.
What is $AIRFLOW_HOME?
$AIRFLOW_HOMEis a location that contains all configuration files, DAGs, plugins, and task logs. It is an environment variable set to
/usr/lib/airflowfor all machine users.
Where can I find Airflow Configuration files?
Configuration file is present at “$AIRFLOW_HOME/airflow.cfg”.
Where can I find Airflow DAGs?
The DAGs’ configuration file is available in the
Where can I find Airflow task logs?
The task log configuration file is available in
Where can I find Airflow plugins?
The configuration file is available in
How do I restart Airflow Services?
You can do start/stop/restart actions on an Airflow service and the commands used for each service are given below:
sudo monit <action> schedulerfor Airflow Scheduler.
sudo monit <action> webserverfor Airflow Webserver.
sudo monit <action> workerfor Celery workers. A stop operation gracefully shuts down existing workers. A start operation adds more equivalent number of workers as per the configuration. A restart operation gracefully shuts down existing workers and adds equivalent number of workers as per the configuration.
sudo monit <action> rabbitmqfor RabbitMQ.
How do I invoke Airflow CLI commands within the node?
Airflow is installed inside a virtual environment at the
/usr/lib/virtualenv/python27location. Firstly, activate the virtual envirnoment,
source /usr/lib/virtualenv/python27/bin/activateand run the Airflow command.