Cloudera Docs

Step 5: Configure the Spark Cluster

Additional configuration required while working with the Spark cluster.

  1. In your Cloudera Manager instance, add the following safety-valve settings for Spark cluster mode:
    For SPARK_ON_YARN>GATEWAY role:
    spark-conf/spark-env.sh_client_config_safety_valve
    export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS 
--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=<BCTLS_JARS_DIR>"
    For SPARK3_ON_YARN > GATEWAY role:
    spark3-conf/spark-env.sh_client_config_safety_valve
    export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS 
--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=<BCTLS_JARS_DIR>"
    <BCTLS_JARS_DIR> is the directory containing the SafeLogic bctls and fips core jar files.
  2. For Spark to work correctly on FIPS, add the following safety-valve settings:
    For SPARK_ON_YARN>GATEWAY role:
    spark-conf/spark-defaults.conf_client_config_safety_valve
    spark.yarn.am.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.driver.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.executor.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
    For SPARK3_ON_YARN > GATEWAY role:
    spark3-conf/spark-defaults.conf_client_config_safety_valve
    spark.yarn.am.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.driver.defaultJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.executor.defaultJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true

    Where <BCTLS_JARS_DIR> is the directory containing the SafeLogic bctls and fips core jar files.