Accessing Additional Spark Logs¶
Apart from accessing the logs from the QDS UI, you can also access the following logs, which reside on the cluster, to identify the errors and exceptions in Spark jobs failures:
and it contains the Spark event logs.
- Spark History Server Logs: The
files are stored at
/media/ephemeral0/logs/spark. The Spark history server logs are stored only on the master node of the cluster.
- Spark Event Logs: The
Spark eventlogfiles are stored at
- scheme is the Cloud-specific URI scheme:
oci://for Oracle OCI.
- defloc is the default storage location for the QDS account.
- cluster_id is the cluster ID as shown on the Clusters page of the QDS UI.
- cluster_inst_id is the cluster instance ID. You should contact Qubole Support to obtain the cluster instance ID.