Encrypting and Authenticating Spark Data in Transit¶
Spark supports encryption and authentication of in-transit data. Authentication is done via shared secret; encryption uses the Simple Authentication and Security Layer (SASL). For more information, see this Spark page.
To enable encryption and authentication for a Spark cluster, proceed as follows:
- From the main menu navigate to the Clusters page.
- Choose Edit for the Spark cluster on which you want to enable encryption and authentication.
- In the Hadoop Cluster Settings section, add the following to the Override Hadoop Configuration Variables field:
In the Spark Cluster Settings section, add the following to the Override Spark Configuration field:
spark.authenticate=true spark.authenticate.enableSaslEncryption=true spark.network.sasl.serverAlwaysEncrypt=true
If the cluster is running, restart it to apply these new settings.
All in-transit data will now be encrypted for all Spark jobs running on this cluster.