Encrypting and Authenticating Spark Data in TransitΒΆ

Spark supports encryption and authentication of in-transit data. Authentication is done via shared secret; encryption uses the Simple Authentication and Security Layer (SASL). For more information, see this Spark page.

To enable encryption and authentication for a Spark cluster, proceed as follows:

  1. From the main menu navigate to the Clusters page.
  2. Choose Edit for the Spark cluster on which you want to enable encryption and authentication.
  3. In the Hadoop Cluster Settings section, add the following to the Override Hadoop Configuration Variables field:
spark.authenticate=true

  1. In the Spark Cluster Settings section, add the following to the Override Spark Configuration field:

    spark.authenticate=true
spark.authenticate.enableSaslEncryption=true
spark.network.sasl.serverAlwaysEncrypt=true

  2. If the cluster is running, restart it to apply these new settings.

All in-transit data will now be encrypted for all Spark jobs running on this cluster.