flink Jobhistory

🧙 Questions

使用yarn部署flink作业的时候，当yarn中的flink作业停止，无法查看该作业的具体异常和具体状态。可以通过启动flink的jobhistory功能，实现记录flink作业状态功能

☄️ Ideas

前提

• yarn开启history

修改flink配置

vim /opt/flink/conf/flink-conf.yaml

# 配置history储存日志的路径
jobmanager.archive.fs.dir: hdfs://isxcode:9000/completed-jobs

# 配置history读取日志的hdfs路径，一定要和jobmanager.archive.fs.dir路径保持一直
historyserver.archive.fs.dir: hdfs://isxcode:9000/completed-jobs

# jobhistory访问端口，默认端口号8082
historyserver.web.port: 8082
historyserver.web.address: isxcode


historyserver.archive.fs.refresh-interval: 10000

# historyserver.log.jobmanager.url-pattern: http://my.log-browsing.url/<jobid>

# historyserver.log.taskmanager.url-pattern: http://my.log-browsing.url/<jobid>/<tmid>

启动jobhistory

关闭jobhistory: bin/historyserver.sh stop

cd /opt/flink
bin/historyserver.sh start

使用jps查看服务是否启动

jps
# 10224 HistoryServer

UI

• http://isxcode:8082/#/overview

查看作业状态

• http://isxcode:8082/jobs/${flinkJobId}

• http://isxcode:8082/jobs/${flinkJobId}/exceptions

官方Api

先通过get请求，获取json的字符串，然后构建一个对象，通过Fastjson将String转成可用的对象。

/config
/jobs/overview
/jobs/<jobid>
/jobs/<jobid>/vertices
/jobs/<jobid>/config
/jobs/<jobid>/exceptions
/jobs/<jobid>/accumulators
/jobs/<jobid>/vertices/<vertexid>
/jobs/<jobid>/vertices/<vertexid>/subtasktimes
/jobs/<jobid>/vertices/<vertexid>/taskmanagers
/jobs/<jobid>/vertices/<vertexid>/accumulators
/jobs/<jobid>/vertices/<vertexid>/subtasks/accumulators
/jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>
/jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>
/jobs/<jobid>/vertices/<vertexid>/subtasks/<subtasknum>/attempts/<attempt>/accumulators
/jobs/<jobid>/plan
/jobs/<jobid>/jobmanager/config
/jobs/<jobid>/jobmanager/environment
/jobs/<jobid>/jobmanager/log-url
/jobs/<jobid>/taskmanagers/<taskmanagerid>/log-url

🔗 Links

• flink docs
• flink jobhistory docs