spark HistoryServer

🧙 Questions

spark开启historyServer功能

☄️ Ideas

创建hdfs目录

hadoop fs -mkdir -p /spark/logs/

配置spark-defaults.conf文件

cp /opt/spark/conf/spark-defaults.conf.template /opt/spark/conf/spark-defaults.conf
vim /opt/spark/conf/spark-defaults.conf

直接新增

spark.eventLog.enabled             true
spark.eventLog.dir                 hdfs://isxcode:9000/spark/logs
spark.history.fs.logDirectory      hdfs://isxcode:9000/spark/logs
spark.history.update.interval      2
spark.history.retainedApplications 50
spark.history.ui.port              18080

启动

关闭服务：stop-history-server.sh

start-history-server.sh

测试

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local \
  /opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar \
  10

• http://isxcode:18080

开启隐藏接口

cp /opt/spark/conf/spark-defaults.conf.template /opt/spark/conf/spark-defaults.conf
tee -a /opt/spark/conf/spark-defaults.conff <<-'EOF'
spark.master.rest.enabled        true
spark.master.rest.port           16060
EOF

• http://47.92.145.79:16060/v1/submissions/status/driver-20230612161336-0000

测试

–supervise app运行失败 driver会重新启动一个app

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://isxcode:7077 \
  --deploy-mode cluster \
  /opt/spark/examples/jars/spark-examples_2.12-3.4.0.jar \
  100

spark-submit \
  --status driver-20230612162232-0000 \
  --master spark://isxcode:16060

23/06/12 15:51:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/06/12 15:51:47 INFO SecurityManager: Changing view acls to: ispong
23/06/12 15:51:47 INFO SecurityManager: Changing modify acls to: ispong
23/06/12 15:51:47 INFO SecurityManager: Changing view acls groups to: 
23/06/12 15:51:47 INFO SecurityManager: Changing modify acls groups to: 
23/06/12 15:51:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: ispong; groups with view permissions: EMPTY; users with modify permissions: ispong; groups with modify permissions: EMPTY
23/06/12 15:51:47 INFO Utils: Successfully started service 'driverClient' on port 33370.
23/06/12 15:51:47 INFO TransportClientFactory: Successfully created connection to isxcode/172.16.215.104:7077 after 39 ms (0 ms spent in bootstraps)
23/06/12 15:51:48 INFO ClientEndpoint: ... waiting before polling master for driver state
23/06/12 15:51:48 INFO ClientEndpoint: Driver successfully submitted as driver-20230612155148-0000
23/06/12 15:51:53 INFO ClientEndpoint: State of driver-20230612155148-0000 is RUNNING
23/06/12 15:51:53 INFO ClientEndpoint: Driver running on 172.16.215.104:41221 (worker-20230612155117-172.16.215.104-41221)
23/06/12 15:51:53 INFO ClientEndpoint: spark-submit not configured to wait for completion, exiting spark-submit JVM.
23/06/12 15:51:53 INFO ShutdownHookManager: Shutdown hook called
23/06/12 15:51:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-268a86be-1687-4737-ac96-e5ea9e4c9bde

🔗 Links

• spark website