Accessing Additional Spark Logs

Apart from accessing the logs from the QDS UI, you can also access the following logs, which reside on the cluster, to identify the errors and exceptions in Spark jobs failures:

and it contains the Spark event logs.

  • Spark History Server Logs: The spark-yarn-org.apache.spark.deploy.history.HistoryServer-1-localhost.localdomain.log

files are stored at /media/ephemeral0/logs/spark. The Spark history server logs are stored only on the coordinator node of the cluster.

  • Spark Event Logs: The Spark eventlog files are stored at

<scheme><defloc>/logs/hadoop/<cluster_id>/<cluster_inst_id>/spark-eventlogs where:

  • scheme is the Cloud-specific URI scheme: s3:// for AWS; wasb:// or adl:// or abfs[s] for Azure; oci:// for Oracle OCI.

  • defloc is the default storage location for the QDS account.

  • cluster_id is the cluster ID as shown on the Clusters page of the QDS UI.

  • cluster_inst_id is the cluster instance ID. You should contact Qubole Support to obtain the cluster instance ID.