spark执行参数分析

脚本示例:
/home/qinxike/spark-submit2.sh --deploy-mode cluster --name amazonETL_0.0.1 --conf spark.sql.shuffle.partitions=400 --conf spark.default.parallelism=400 --conf spark.memory.storageFraction=0.1 --conf spark.memory.fraction=0.75 --conf spark.yarn.driver.memoryOverhead=4096 --conf spark.yarn.executor.memoryOverhead=4096 --executor-memory 20g --driver-memory 10g --num-executors 44 --executor-cores 4 /home/qinxike/amazon-etl-v0.0.1-driver.jar -b etl -g maxstep=1 -g jointotal=0 -g test=y -a amazonETL
spark占用内存计算:
总内存= (executor-memory+spark.yarn.executor.memoryOverhead)*num-executors + (driver-memory + spark.yarn.driver.memoryOverhead)
      = (20+4)*44 + (10+4)
      = 1070(G)