HistoryServer
a inner join with one empty side and one large side, the plan would finish and the large side is still running.
- Use SparkListenerSQLExecutionEnd to track final SQL status in UI
https://issues.apache.org/jira/browse/SPARK-40834
Spark HistoryServer 3.4 以后修复了这个问题
JsonProtocol
JsonProtocol 类为 HistoryServer 提供解析 eventLog 的基本工具类库。
other 选项为其提供了 基本反射功能,可以解决各类 event 事件
FsHistoryProvider
override def start(): Unit = {
initThread = initialize()
}
private[history] def initialize(): Thread = {
if (!isFsInSafeMode()) {
startPolling()
null
} else {
startSafeModeCheckThread(None)
}
}