Spark 之 HistoryServer and FsHistoryProvider

zhixingheyi_tian2024-11-02 9:06

HistoryServer

a inner join with one empty side and one large side, the plan would finish and the large side is still running.

Use SparkListenerSQLExecutionEnd to track final SQL status in UI
https://issues.apache.org/jira/browse/SPARK-40834

Spark HistoryServer 3.4 以后修复了这个问题

JsonProtocol

JsonProtocol 类为 HistoryServer 提供解析 eventLog 的基本工具类库。

other 选项为其提供了基本反射功能，可以解决各类 event 事件

FsHistoryProvider

复制代码

  override def start(): Unit = {
    initThread = initialize()
  }

  private[history] def initialize(): Thread = {
    if (!isFsInSafeMode()) {
      startPolling()
      null
    } else {
      startSafeModeCheckThread(None)
    }
  }