[Spark] TaskMetrics指标收集

可惜我是水瓶座__2025-10-26 19:33

Executor.TaskRunner.run --> serializedResult
这里排除一些零值，序列化后发送

scala 复制代码

val accumUpdates = if (RESULT_DROP_ZEROES_PREFIXES.nonEmpty) {
  task.collectAccumulatorUpdates()
      .filterNot(acc => RESULT_DROP_ZEROES_PREFIXES
          .exists(acc.getClass.getName.startsWith(_)) && acc.isZero)
} else {
  task.collectAccumulatorUpdates()
}
val serializedDirectResult = ser.serialize(directResult)

CoarseGrainedExecutorBackend.statusUpdate --> data
CoarseGrainedSchedulerBackend.receive --> data.value
TaskSchedulerImpl.statusUpdate --> serializedData: ByteBuffer
TaskResultGetter.enqueueSuccessfulTask
反序列化后得到 result，是一个TaskResult

scala 复制代码

val (result, size) = serializer.get().deserialize[TaskResult[_]](serializedData)

TaskSchedulerImpl.handleSuccessfulTask --> taskResult
TaskSetManager.handleSuccessfulTask --> result.accumUpdates
DAGScheduler.taskEnded --> accumUpdates
发送Task完成消息到eventProcessLoop，等待处理

scala 复制代码

eventProcessLoop.post(CompletionEvent(task, reason, result, accumUpdates, metricPeaks, taskInfo))

DAGScheduler.handleTaskCompletion --> event
DAGScheduler.postTaskEnd
这里通过fromAccumulators得到 taskMertics，最后发送 TaskEnd 事件

scala 复制代码

val taskMetrics: TaskMetrics = TaskMetrics.fromAccumulators(event.accumUpdates)

listenerBus.post(SparkListenerTaskEnd(event.task.stageId, event.task.stageAttemptId,
      Utils.getFormattedClassName(event.task), event.reason, event.taskInfo,
      new ExecutorMetrics(event.metricPeaks), taskMetrics))