目录
Linux Version：linux-5.4.239
一、相关概念
在单核操作系统中，操作系统只需管理一个CPU，当系统有任务需要执行时，所有的任务会在该CPU的就绪队列上进行排队，然后调度器会选择一个合适的任务执行。当就绪队列上的所有任务都执行完成后，CPU就会执行idle进程而进入空闲状态，但由于idle进程的优先级最低，一旦有其它任务进入就绪队列，就会抢占idle进程转去执行其它任务。CPU就是在不同的任务间切换，实现整个系统的运行。其基本架构如下：
但是在引入SMP(Symmetric Multi Processing)之后，操作系统需要同时管理多个CPU，每个CPU拥有自己的就绪队列和idle进程，并可以独立地执行调度操作。当CPU之间的负载出现不均衡时，可以由负载均衡模块在它们之间迁移任务。相应的其基本架构如下：
如果只考虑让操作系统能够正常运行，那只需要一个CPU即可，当操作系统任务不多或者系统需要Suspend时，就可以将多余的CPU从系统中下线，以减少系统功耗，因此在SMP系统中引入了CPU Hotplug的功能，以实现动态地关闭或者打开 Non-Boot CPU。
CPU hotplug 目前主要有以下两个应用场景：
（1）用户通过sysfs接口手动开启或关闭某个cpu
echo 0 > /sys/devices/system/cpu/cpu1/online # 关闭CPU1
echo 1 > /sys/devices/system/cpu/cpu1/online # 打开CPU1
（2）作为系统休眠唤醒流程的一部分，当系统需要休眠时可以前先关闭所有的Secondary CPU，当系统被唤醒后再重新开启。
suspend_enter
-> suspend_disable_secondary_cpus
-> freeze_secondary_cpus
-> __freeze_secondary_cpus
-> _cpu_down(cpu, 1, CPUHP_OFFLINE)
二、基本原理
CPU Online相当于把一个关闭的CPU重新上电运行，因此在开启时需要先为该CPU设置内核的入口地址，然后上电运行，当CPU执行一系列的初始化流程之后，最终加入内核的调度系统参与任务调度。而 CPU Offline则是将一个当前正在执行任务的CPU下电，因为运行中的CPU和系统中很多模块都有联系，所以如果要下线一个CPU，需要先将CPU与相关模块进行分离 ( 解耦)，如将该 CPU 就绪队列上的所有任务，中断等都迁移到其它CPU上去。
当CPU与系统中的相关模块分离之后，此时就可以通过调用 cpu_psci_cpu_die -> psci_cpu_off 完成 CPU 的下电，因此，关闭一个CPU的关键就是分离与相关模块之间的联系。
由于各个模块的作用不同，因此打开或关闭的时间点也有所不同，因此kernel为 CPU Hotplug实现了一个状态机，该状态机在不同阶段执行不同的回调函数，并且这些回调函数在不同阶段的执行由不同的CPU负责，通常用来开启或者关闭其它 CPU 的 CPU被称为 Boot Processor （BP）,被 BP 开启或者关闭的 CPU 称为 Application Processor （AP）。
cpu hotplug状态机定义如下：
objectivec
enum cpuhp_state {
CPUHP_INVALID = -1,
CPUHP_OFFLINE = 0,
CPUHP_CREATE_THREADS,
CPUHP_PERF_PREPARE,
......
CPUHP_AP_ACTIVE,
CPUHP_ONLINE,
};该状态机以CPUHP_OFFLINE开始，并以CPUHP_ONLINE结束，其中每个状态都包含CPU上下线需要执行的操作。
当上线一个CPU时，其初始状态为CPUHP_OFFLINE，如果最终成功执行到CPUHP_ONLINE状态，则该CPU成功上线。其相应的状态转换关系如下图所示：
当下线一个CPU时，其初始状态为CPUHP_ONLINE，如果最终成功执行到CPUHP_OFFLINE状态，则CPU成功下线。其相应的状态转换关系如下图所示：
状态机中的每个状态都以cpuhp_step结构体表示，该结构体包含cpu online和offline时需要执行的回调函数。其定义如下：
objectivec
/**
* cpuhp_step - Hotplug state machine step
* @name: Name of the step
* @startup: Startup function of the step
* @teardown: Teardown function of the step
* @cant_stop: Bringup/teardown can't be stopped at this step
*/
struct cpuhp_step {
const char *name;
union {
int (*single)(unsigned int cpu);
int (*multi)(unsigned int cpu,
struct hlist_node *node);
} startup; // cpu online时需要执行的回调
union {
int (*single)(unsigned int cpu);
int (*multi)(unsigned int cpu,
struct hlist_node *node);
} teardown; // cpu offline时需要执行的回调
struct hlist_head list;
bool cant_stop;
bool multi_instance;
};cpuhp_step 结构体的初始化如下：
objectivec
/* Boot processor state steps */
static struct cpuhp_step cpuhp_hp_states[] = {
[CPUHP_OFFLINE] = {
.name = "offline",
.startup.single = NULL,
.teardown.single = NULL,
},
#ifdef CONFIG_SMP
[CPUHP_CREATE_THREADS]= {
.name = "threads:prepare",
.startup.single = smpboot_create_threads,
.teardown.single = NULL,
.cant_stop = true,
},
[CPUHP_PERF_PREPARE] = {
.name = "perf:prepare",
.startup.single = perf_event_init_cpu,
.teardown.single = perf_event_exit_cpu,
},
[CPUHP_RANDOM_PREPARE] = {
.name = "random:prepare",
.startup.single = random_prepare_cpu,
.teardown.single = NULL,
},
[CPUHP_WORKQUEUE_PREP] = {
.name = "workqueue:prepare",
.startup.single = workqueue_prepare_cpu,
.teardown.single = NULL,
},
[CPUHP_HRTIMERS_PREPARE] = {
.name = "hrtimers:prepare",
.startup.single = hrtimers_prepare_cpu,
.teardown.single = hrtimers_dead_cpu,
},
[CPUHP_SMPCFD_PREPARE] = {
.name = "smpcfd:prepare",
.startup.single = smpcfd_prepare_cpu,
.teardown.single = smpcfd_dead_cpu,
},
[CPUHP_RELAY_PREPARE] = {
.name = "relay:prepare",
.startup.single = relay_prepare_cpu,
.teardown.single = NULL,
},
[CPUHP_SLAB_PREPARE] = {
.name = "slab:prepare",
.startup.single = slab_prepare_cpu,
.teardown.single = slab_dead_cpu,
},
[CPUHP_RCUTREE_PREP] = {
.name = "RCU/tree:prepare",
.startup.single = rcutree_prepare_cpu,
.teardown.single = rcutree_dead_cpu,
},
/*
* On the tear-down path, timers_dead_cpu() must be invoked
* before blk_mq_queue_reinit_notify() from notify_dead(),
* otherwise a RCU stall occurs.
*/
[CPUHP_TIMERS_PREPARE] = {
.name = "timers:prepare",
.startup.single = timers_prepare_cpu,
.teardown.single = timers_dead_cpu,
},
/* Kicks the plugged cpu into life */
[CPUHP_BRINGUP_CPU] = {
.name = "cpu:bringup",
.startup.single = bringup_cpu,
.teardown.single = finish_cpu,
.cant_stop = true,
},
/* Final state before CPU kills itself */
[CPUHP_AP_IDLE_DEAD] = {
.name = "idle:dead",
},
/*
* Last state before CPU enters the idle loop to die. Transient state
* for synchronization.
*/
[CPUHP_AP_OFFLINE] = {
.name = "ap:offline",
.cant_stop = true,
},
/* First state is scheduler control. Interrupts are disabled */
[CPUHP_AP_SCHED_STARTING] = {
.name = "sched:starting",
.startup.single = sched_cpu_starting,
.teardown.single = sched_cpu_dying,
},
[CPUHP_AP_RCUTREE_DYING] = {
.name = "RCU/tree:dying",
.startup.single = NULL,
.teardown.single = rcutree_dying_cpu,
},
[CPUHP_AP_SMPCFD_DYING] = {
.name = "smpcfd:dying",
.startup.single = NULL,
.teardown.single = smpcfd_dying_cpu,
},
/* Entry state on starting. Interrupts enabled from here on. Transient
* state for synchronsization */
[CPUHP_AP_ONLINE] = {
.name = "ap:online",
},
/*
* Handled on controll processor until the plugged processor manages
* this itself.
*/
[CPUHP_TEARDOWN_CPU] = {
.name = "cpu:teardown",
.startup.single = NULL,
.teardown.single = takedown_cpu,
.cant_stop = true,
},
/* Handle smpboot threads park/unpark */
[CPUHP_AP_SMPBOOT_THREADS] = {
.name = "smpboot/threads:online",
.startup.single = smpboot_unpark_threads,
.teardown.single = smpboot_park_threads,
},
[CPUHP_AP_IRQ_AFFINITY_ONLINE] = {
.name = "irq/affinity:online",
.startup.single = irq_affinity_online_cpu,
.teardown.single = NULL,
},
[CPUHP_AP_PERF_ONLINE] = {
.name = "perf:online",
.startup.single = perf_event_init_cpu,
.teardown.single = perf_event_exit_cpu,
},
[CPUHP_AP_WATCHDOG_ONLINE] = {
.name = "lockup_detector:online",
.startup.single = lockup_detector_online_cpu,
.teardown.single = lockup_detector_offline_cpu,
},
[CPUHP_AP_WORKQUEUE_ONLINE] = {
.name = "workqueue:online",
.startup.single = workqueue_online_cpu,
.teardown.single = workqueue_offline_cpu,
},
[CPUHP_AP_RANDOM_ONLINE] = {
.name = "random:online",
.startup.single = random_online_cpu,
.teardown.single = NULL,
},
[CPUHP_AP_RCUTREE_ONLINE] = {
.name = "RCU/tree:online",
.startup.single = rcutree_online_cpu,
.teardown.single = rcutree_offline_cpu,
},
#endif
/*
* The dynamically registered state space is here
*/
#ifdef CONFIG_SMP
/* Last state is scheduler control setting the cpu active */
[CPUHP_AP_ACTIVE] = {
.name = "sched:active",
.startup.single = sched_cpu_activate,
.teardown.single = sched_cpu_deactivate,
},
#endif
/* CPU is fully up and running. */
[CPUHP_ONLINE] = {
.name = "online",
.startup.single = NULL,
.teardown.single = NULL,
},
};三、代码分析
1、CPU_ON
根据前面的分析，启动一个 CPU 其实就是将一个 CPU 的状态从 CPUHP_OFFLINE 转换到 CPUHP_ONLINE ，在不同的阶段由不同的 CPU 执行。
如上图所示，当需要启动一个 CPU 时，BP 会先将 AP 的状态转换为 CPUHP_BRINGUP_CPU ，然后执行 bringup_cpu 回调函数，在回调函数中调用 psci_cpu_on 通过smc陷入到 ATF 进行开启 AP，当 AP 执行到 secondary_start_kernel 函数时会告诉 BP 自己已经启动了，让 BP 继续运行，当 AP 在进入 idle 之前还会通知 BP 自己启动到哪里了，这时 BP 就会 kick AP 的 hotplug 线程（cpuhp_thread_fun）继续执行状态机中后面的回调，直到 AP 的状态变成 CPUHP_ONLINE 为止，当 AP 的状态达到 CPUHP_ONLINE 时，AP 也就完全启动了。
CPU_ON 执行log：
注意：
1、本测试平台为 qemu，所以最后是 hvc 陷入到 hypervisor，而不是smc陷入到 ATF。
2、func 表示本行打印所在函数，line 表示行数，st->state 表示CPU的当前状态，CPU表示执行这一行代码的 CPU。
objectivec
# echo 1 > /sys/devices/system/cpu/cpu3/online
[ 150.887484] func: cpu_subsys_online line: 53, CPU: 0
[ 150.887634] func: _cpu_up line: 1219, CPU: 0
[ 150.897642] func: _cpu_up: before cpuhp_up_callbacks line: 1267, CPU: 0
[ 150.897804] func: cpuhp_up_callbacks, st->state: 1, line: 628, CPU: 0
[ 150.897817] func: cpuhp_up_callbacks, st->state: 2, line: 628, CPU: 0
[ 150.897958] func: cpuhp_up_callbacks, st->state: 3, line: 628, CPU: 0
[ 150.898096] func: cpuhp_up_callbacks, st->state: 4, line: 628, CPU: 0
[ 150.898231] func: cpuhp_up_callbacks, st->state: 5, line: 628, CPU: 0
[ 150.898364] func: cpuhp_up_callbacks, st->state: 6, line: 628, CPU: 0
[ 150.898490] func: cpuhp_up_callbacks, st->state: 7, line: 628, CPU: 0
[ 150.898618] func: cpuhp_up_callbacks, st->state: 8, line: 628, CPU: 0
[ 150.898758] func: cpuhp_up_callbacks, st->state: 9, line: 628, CPU: 0
[ 150.898890] func: cpuhp_up_callbacks, st->state: 10, line: 628, CPU: 0
[ 150.899012] func: cpuhp_up_callbacks, st->state: 11, line: 628, CPU: 0
[ 150.899136] func: cpuhp_up_callbacks, st->state: 12, line: 628, CPU: 0
[ 150.899261] func: cpuhp_up_callbacks, st->state: 13, line: 628, CPU: 0
[ 150.899387] func: cpuhp_up_callbacks, st->state: 14, line: 628, CPU: 0
[ 150.899517] func: cpuhp_up_callbacks, st->state: 15, line: 628, CPU: 0
[ 150.899649] func: cpuhp_up_callbacks, st->state: 16, line: 628, CPU: 0
[ 150.899781] func: cpuhp_up_callbacks, st->state: 17, line: 628, CPU: 0
[ 150.899911] func: cpuhp_up_callbacks, st->state: 18, line: 628, CPU: 0
[ 150.900043] func: cpuhp_up_callbacks, st->state: 19, line: 628, CPU: 0
[ 150.900176] func: cpuhp_up_callbacks, st->state: 20, line: 628, CPU: 0
[ 150.900306] func: cpuhp_up_callbacks, st->state: 21, line: 628, CPU: 0
[ 150.900441] func: cpuhp_up_callbacks, st->state: 22, line: 628, CPU: 0
[ 150.900571] func: cpuhp_up_callbacks, st->state: 23, line: 628, CPU: 0
[ 150.900701] func: cpuhp_up_callbacks, st->state: 24, line: 628, CPU: 0
[ 150.900827] func: cpuhp_up_callbacks, st->state: 25, line: 628, CPU: 0
[ 150.900956] func: cpuhp_up_callbacks, st->state: 26, line: 628, CPU: 0
[ 150.901088] func: cpuhp_up_callbacks, st->state: 27, line: 628, CPU: 0
[ 150.901221] func: cpuhp_up_callbacks, st->state: 28, line: 628, CPU: 0
[ 150.901352] func: cpuhp_up_callbacks, st->state: 29, line: 628, CPU: 0
[ 150.901489] func: cpuhp_up_callbacks, st->state: 30, line: 628, CPU: 0
[ 150.904508] func: cpuhp_up_callbacks, st->state: 31, line: 628, CPU: 0
[ 150.904657] func: cpuhp_up_callbacks, st->state: 32, line: 628, CPU: 0
[ 150.904805] func: cpuhp_up_callbacks, st->state: 33, line: 628, CPU: 0
[ 150.904945] func: cpuhp_up_callbacks, st->state: 34, line: 628, CPU: 0
[ 150.905082] func: cpuhp_up_callbacks, st->state: 35, line: 628, CPU: 0
[ 150.905220] func: cpuhp_up_callbacks, st->state: 36, line: 628, CPU: 0
[ 150.905363] func: cpuhp_up_callbacks, st->state: 37, line: 628, CPU: 0
[ 150.905517] func: cpuhp_up_callbacks, st->state: 38, line: 628, CPU: 0
[ 150.906525] func: cpuhp_up_callbacks, st->state: 39, line: 628, CPU: 0
[ 150.906674] func: cpuhp_up_callbacks, st->state: 40, line: 628, CPU: 0
[ 150.906814] func: cpuhp_up_callbacks, st->state: 41, line: 628, CPU: 0
[ 150.906949] func: cpuhp_up_callbacks, st->state: 42, line: 628, CPU: 0
[ 150.907108] func: cpuhp_up_callbacks, st->state: 43, line: 628, CPU: 0
[ 150.907242] func: cpuhp_up_callbacks, st->state: 44, line: 628, CPU: 0
[ 150.907376] func: cpuhp_up_callbacks, st->state: 45, line: 628, CPU: 0
[ 150.907512] func: cpuhp_up_callbacks, st->state: 46, line: 628, CPU: 0
[ 150.907649] func: cpuhp_up_callbacks, st->state: 47, line: 628, CPU: 0
[ 150.907784] func: cpuhp_up_callbacks, st->state: 48, line: 628, CPU: 0
[ 150.907917] func: cpuhp_up_callbacks, st->state: 49, line: 628, CPU: 0
[ 150.908048] func: cpuhp_up_callbacks, st->state: 50, line: 628, CPU: 0
[ 150.908179] func: cpuhp_up_callbacks, st->state: 51, line: 628, CPU: 0
[ 150.908308] func: cpuhp_up_callbacks, st->state: 52, line: 628, CPU: 0
[ 150.908437] func: cpuhp_up_callbacks, st->state: 53, line: 628, CPU: 0
[ 150.908571] func: cpuhp_up_callbacks, st->state: 54, line: 628, CPU: 0
[ 150.908705] func: cpuhp_up_callbacks, st->state: 55, line: 628, CPU: 0
[ 150.908892] func: cpuhp_up_callbacks, st->state: 56, line: 628, CPU: 0
[ 150.909027] func: cpuhp_up_callbacks, st->state: 57, line: 628, CPU: 0
[ 150.909157] func: cpuhp_up_callbacks, st->state: 58, line: 628, CPU: 0
[ 150.909288] func: cpuhp_up_callbacks, st->state: 59, line: 628, CPU: 0
[ 150.909419] func: cpuhp_up_callbacks, st->state: 60, line: 628, CPU: 0
[ 150.911541] func: cpuhp_up_callbacks, st->state: 61, line: 628, CPU: 0
[ 150.911692] func: cpuhp_up_callbacks, st->state: 62, line: 628, CPU: 0
[ 150.911828] func: cpuhp_up_callbacks, st->state: 63, line: 628, CPU: 0
[ 150.911960] func: cpuhp_up_callbacks, st->state: 64, line: 628, CPU: 0
[ 150.912096] func: cpuhp_up_callbacks, st->state: 65, line: 628, CPU: 0
[ 150.912227] func: cpuhp_up_callbacks, st->state: 66, line: 628, CPU: 0
[ 150.912360] func: cpuhp_up_callbacks, st->state: 67, line: 628, CPU: 0
[ 150.912490] func: cpuhp_up_callbacks, st->state: 68, line: 628, CPU: 0
[ 150.912629] func: cpuhp_up_callbacks, st->state: 69, line: 628, CPU: 0
[ 150.912770] func: cpuhp_up_callbacks, st->state: 70, line: 628, CPU: 0
[ 150.912904] func: cpuhp_up_callbacks, st->state: 71, line: 628, CPU: 0
[ 150.913040] func: cpuhp_up_callbacks, st->state: 72, line: 628, CPU: 0
[ 150.913176] func: cpuhp_up_callbacks, st->state: 73, line: 628, CPU: 0
[ 150.913311] func: cpuhp_up_callbacks, st->state: 74, line: 628, CPU: 0
[ 150.913447] func: cpuhp_up_callbacks, st->state: 75, line: 628, CPU: 0
[ 150.915031] func: cpuhp_up_callbacks, st->state: 76, line: 628, CPU: 0
[ 150.915174] func: cpuhp_up_callbacks, st->state: 77, line: 628, CPU: 0
[ 150.915308] func: cpuhp_up_callbacks, st->state: 78, line: 628, CPU: 0
[ 150.915440] func: cpuhp_up_callbacks, st->state: 79, line: 628, CPU: 0
[ 150.915575] func: cpuhp_up_callbacks, st->state: 80, line: 628, CPU: 0
[ 150.915712] func: cpuhp_up_callbacks, st->state: 81, line: 628, CPU: 0
[ 150.915847] func: cpuhp_up_callbacks, st->state: 82, line: 628, CPU: 0
[ 150.915980] func: cpuhp_up_callbacks, st->state: 83, line: 628, CPU: 0
[ 150.916111] func: cpuhp_up_callbacks, st->state: 84, line: 628, CPU: 0
[ 150.916242] func: cpuhp_up_callbacks, st->state: 85, line: 628, CPU: 0
[ 150.916368] func: cpuhp_up_callbacks, st->state: 86, line: 628, CPU: 0
[ 150.916503] func: cpuhp_up_callbacks, st->state: 87, line: 628, CPU: 0
[ 150.916641] func: bringup_cpu line: 558, CPU: 0
[ 150.916869] func: bringup_cpu: before __cpu_up line: 566, CPU: 0
[ 150.916995] func: __cpu_up line: 106, CPU: 0
[ 150.917091] psci: func: psci_cpu_on line: 199, entry_point: 40c9e1e0, CPU: 0
[ 150.917233] psci: func: __invoke_psci_fn_hvc line: 136, CPU: 0
[ 150.917471] Detected VIPT I-cache on CPU3
[ 150.917602] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000
[ 150.918077] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[ 150.918242] func: cpu_startup_entry, line: 365
[ 150.918344] func: cpu_startup_entry, before cpuhp_online_idle function line: 367, CPU: 3
[ 150.918530] func: cpu_startup_entry, after cpuhp_online_idle function line: 369, CPU: 3
[ 150.920524] func: __cpu_up: after wait_for_completion_timeout line: 127, CPU: 0
[ 150.920693] func: bringup_cpu: after __cpu_up line: 569, CPU: 0
[ 150.920819] func: bringup_cpu: before bringup_wait_for_ap line: 575, CPU: 0
[ 150.920961] func: bringup_wait_for_ap: before wait_for_ap_thread line: 525, CPU: 0
[ 150.921115] func: bringup_wait_for_ap: after wait_for_ap_thread line: 530, CPU: 0
[ 150.921278] func: bringup_wait_for_ap: before cpuhp_kick_ap line: 548, CPU: 0
[ 150.921497] func: cpuhp_thread_fun st->state: 140, line: 696, CPU: 3
[ 150.921522] func: cpuhp_thread_fun st->state: 141, line: 696, CPU: 3
[ 150.921718] func: cpuhp_thread_fun st->state: 142, line: 696, CPU: 3
[ 150.921854] func: cpuhp_thread_fun st->state: 143, line: 696, CPU: 3
[ 150.921977] func: cpuhp_thread_fun st->state: 144, line: 696, CPU: 3
[ 150.922085] func: cpuhp_thread_fun st->state: 145, line: 696, CPU: 3
[ 150.922194] func: cpuhp_thread_fun st->state: 146, line: 696, CPU: 3
[ 150.922300] func: cpuhp_thread_fun st->state: 147, line: 696, CPU: 3
[ 150.922399] func: cpuhp_thread_fun st->state: 148, line: 696, CPU: 3
[ 150.922515] func: cpuhp_thread_fun st->state: 149, line: 696, CPU: 3
[ 150.922621] func: cpuhp_thread_fun st->state: 150, line: 696, CPU: 3
[ 150.922725] func: cpuhp_thread_fun st->state: 151, line: 696, CPU: 3
[ 150.922832] func: cpuhp_thread_fun st->state: 152, line: 696, CPU: 3
[ 150.922933] func: cpuhp_thread_fun st->state: 153, line: 696, CPU: 3
[ 150.923051] func: cpuhp_thread_fun st->state: 154, line: 696, CPU: 3
[ 150.923166] func: cpuhp_thread_fun st->state: 155, line: 696, CPU: 3
[ 150.923269] func: cpuhp_thread_fun st->state: 156, line: 696, CPU: 3
[ 150.923380] func: cpuhp_thread_fun st->state: 157, line: 696, CPU: 3
[ 150.923501] func: cpuhp_thread_fun st->state: 158, line: 696, CPU: 3
[ 150.923604] func: cpuhp_thread_fun st->state: 159, line: 696, CPU: 3
[ 150.923723] func: cpuhp_thread_fun st->state: 160, line: 696, CPU: 3
[ 150.923831] func: cpuhp_thread_fun st->state: 161, line: 696, CPU: 3
[ 150.923953] func: cpuhp_thread_fun st->state: 162, line: 696, CPU: 3
[ 150.924064] func: cpuhp_thread_fun st->state: 163, line: 696, CPU: 3
[ 150.924175] func: cpuhp_thread_fun st->state: 164, line: 696, CPU: 3
[ 150.924288] func: cpuhp_thread_fun st->state: 165, line: 696, CPU: 3
[ 150.924395] func: cpuhp_thread_fun st->state: 166, line: 696, CPU: 3
[ 150.924508] func: cpuhp_thread_fun st->state: 167, line: 696, CPU: 3
[ 150.924619] func: cpuhp_thread_fun st->state: 168, line: 696, CPU: 3
[ 150.924735] func: cpuhp_thread_fun st->state: 169, line: 696, CPU: 3
[ 150.924844] func: cpuhp_thread_fun st->state: 170, line: 696, CPU: 3
[ 150.925051] func: cpuhp_thread_fun st->state: 171, line: 696, CPU: 3
[ 150.925168] func: cpuhp_thread_fun st->state: 172, line: 696, CPU: 3
[ 150.925287] func: cpuhp_thread_fun st->state: 173, line: 696, CPU: 3
[ 150.925401] func: cpuhp_thread_fun st->state: 174, line: 696, CPU: 3
[ 150.925511] func: cpuhp_thread_fun st->state: 175, line: 696, CPU: 3
[ 150.925685] func: cpuhp_thread_fun st->state: 176, line: 696, CPU: 3
[ 150.925816] func: cpuhp_thread_fun st->state: 177, line: 696, CPU: 3
[ 150.925983] func: cpuhp_thread_fun st->state: 178, line: 696, CPU: 3
[ 150.926103] func: cpuhp_thread_fun st->state: 179, line: 696, CPU: 3
[ 150.926224] func: cpuhp_thread_fun st->state: 180, line: 696, CPU: 3
[ 150.926367] func: cpuhp_thread_fun st->state: 181, line: 696, CPU: 3
[ 150.926487] func: cpuhp_thread_fun st->state: 182, line: 696, CPU: 3
[ 150.926611] func: cpuhp_thread_fun st->state: 183, line: 696, CPU: 3
[ 150.926722] func: cpuhp_thread_fun st->state: 184, line: 696, CPU: 3
[ 150.926830] func: cpuhp_thread_fun st->state: 185, line: 696, CPU: 3
[ 150.926928] func: cpuhp_thread_fun st->state: 186, line: 696, CPU: 3
[ 150.927028] func: cpuhp_thread_fun st->state: 187, line: 696, CPU: 3
[ 150.927145] func: cpuhp_thread_fun st->state: 188, line: 696, CPU: 3
[ 150.927259] func: cpuhp_thread_fun st->state: 189, line: 696, CPU: 3
[ 150.927360] func: cpuhp_thread_fun st->state: 190, line: 696, CPU: 3
[ 150.927468] func: cpuhp_thread_fun st->state: 191, line: 696, CPU: 3
[ 150.927581] func: cpuhp_thread_fun st->state: 192, line: 696, CPU: 3
[ 150.927687] func: cpuhp_thread_fun st->state: 193, line: 696, CPU: 3
[ 150.927789] func: cpuhp_thread_fun st->state: 194, line: 696, CPU: 3
[ 150.927902] func: cpuhp_thread_fun st->state: 195, line: 696, CPU: 3
[ 150.928023] func: cpuhp_thread_fun st->state: 196, line: 696, CPU: 3
[ 150.928132] func: cpuhp_thread_fun st->state: 197, line: 696, CPU: 3
[ 150.928251] func: cpuhp_thread_fun st->state: 198, line: 696, CPU: 3
[ 150.928357] func: cpuhp_thread_fun st->state: 199, line: 696, CPU: 3
[ 150.928469] func: cpuhp_thread_fun st->state: 200, line: 696, CPU: 3
[ 150.928580] func: cpuhp_thread_fun st->state: 201, line: 696, CPU: 3
[ 150.928689] func: cpuhp_thread_fun st->state: 202, line: 696, CPU: 3
[ 150.928798] func: cpuhp_thread_fun st->state: 203, line: 696, CPU: 3
[ 150.928914] func: cpuhp_thread_fun st->state: 204, line: 696, CPU: 3
[ 150.929018] func: cpuhp_thread_fun st->state: 205, line: 696, CPU: 3
[ 150.929125] func: cpuhp_thread_fun st->state: 206, line: 696, CPU: 3
[ 150.929232] func: cpuhp_thread_fun st->state: 207, line: 696, CPU: 3
[ 150.929389] func: cpuhp_thread_fun st->state: 208, line: 696, CPU: 3
[ 150.929503] func: cpuhp_thread_fun before complete_ap_thread st->state: 209 , line: 744, CPU: 3
[ 150.929768] func: bringup_cpu: after bringup_wait_for_ap line: 577, CPU: 0
[ 150.930102] func: _cpu_up: after cpuhp_up_callbacks line: 1269, CPU: 0
# 可以看到 CPU_ON 最开始是由 BP 调用状态机中的回调函数，当调用到第 87 （CPUHP_BRINGUP_CPU）个状态时，调用 bringup_cpu 回调进行开核，当 AP 启动之后会自己调用后面状态的回调，直至 AP 状态等于 209 （CPUHP_ONLINE）。
2、CPU_OFF
CPU_OFF 的执行过程刚好和 CPU_ON 相反，当一个 CPU 的状态从 CPUHP_ONLINE 变成 CPUHP_OFFLINE 时，说明这个 CPU 已经被关闭了。
如上图所示，当需要关闭一个 CPU 时，BP 会先调用 cpuhp_kick_ap_work 函数让 AP 的 hotplug 线程去执行状态转换，会从 CPUHP_ONLINE 转换成 CPUHP_TEARDOWN_CPU，然后执行 takedown_cpu 回调函数，在该回调函数中会通过 stop_machine_cpuslocked 函数将 AP 设置为 离线状态，然后在 AP 的 idle 线程中会调用 cpu_is_offline 函数来检测 AP 是否已经离线，如果已经离线则会调用 cpuhp_report_idle_dead 函数告诉 BP，BP 会在__cpu_die函数中调用cpu_wait_death函数等待 AP 死亡，这时 AP 就会调用 arch_cpu_idle_dead 函数，告诉 BP 这个cpu现在可以安全地处理了，然后调用 psci_cpu_off 通过 smc 陷入到 ATF 中完成 AP 的下电操作，当 takedown_cpu 函数执行完成后 BP 会回到 cpuhp_down_callbacks 函数中继续调用 AP 后续状态的回调函数，直到 AP 状态变成 CPUHP_OFFLINE 为止，当 AP 状态变成 CPUHP_OFFLINE 时，AP 也就完全关闭了。
CPU_OFF 执行log：
注意：
1、本测试平台为 qemu，所以最后是 hvc 陷入到 hypervisor，而不是smc陷入到 ATF。
2、func 表示本行打印所在函数，line 表示行数，st->state 表示CPU的当前状态，CPU表示执行这一行代码的 CPU。
objectivec
# echo 0 > /sys/devices/system/cpu/cpu3/online
[ 105.166968] func: cpu_subsys_offline line: 72, CPU: 3
[ 105.167080] func: _cpu_down line: 1076, CPU: 3
[ 105.181625] func: _cpu_down: before cpuhp_kick_ap_work line: 1094, CPU: 3
[ 105.181770] func: cpuhp_thread_fun st->state: 209, line: 696, CPU: 3
[ 105.181778] func: cpuhp_thread_fun st->state: 208, line: 696, CPU: 3
[ 105.213647] func: cpuhp_thread_fun st->state: 207, line: 696, CPU: 3
[ 105.213833] func: cpuhp_thread_fun st->state: 206, line: 696, CPU: 3
[ 105.213993] func: cpuhp_thread_fun st->state: 205, line: 696, CPU: 3
[ 105.214168] func: cpuhp_thread_fun st->state: 204, line: 696, CPU: 3
[ 105.214338] func: cpuhp_thread_fun st->state: 203, line: 696, CPU: 3
[ 105.214444] func: cpuhp_thread_fun st->state: 202, line: 696, CPU: 3
[ 105.214539] func: cpuhp_thread_fun st->state: 201, line: 696, CPU: 3
[ 105.214634] func: cpuhp_thread_fun st->state: 200, line: 696, CPU: 3
[ 105.214730] func: cpuhp_thread_fun st->state: 199, line: 696, CPU: 3
[ 105.214826] func: cpuhp_thread_fun st->state: 198, line: 696, CPU: 3
[ 105.214926] func: cpuhp_thread_fun st->state: 197, line: 696, CPU: 3
[ 105.215021] func: cpuhp_thread_fun st->state: 196, line: 696, CPU: 3
[ 105.215113] func: cpuhp_thread_fun st->state: 195, line: 696, CPU: 3
[ 105.215208] func: cpuhp_thread_fun st->state: 194, line: 696, CPU: 3
[ 105.215304] func: cpuhp_thread_fun st->state: 193, line: 696, CPU: 3
[ 105.215396] func: cpuhp_thread_fun st->state: 192, line: 696, CPU: 3
[ 105.215487] func: cpuhp_thread_fun st->state: 191, line: 696, CPU: 3
[ 105.215577] func: cpuhp_thread_fun st->state: 190, line: 696, CPU: 3
[ 105.215669] func: cpuhp_thread_fun st->state: 189, line: 696, CPU: 3
[ 105.215765] func: cpuhp_thread_fun st->state: 188, line: 696, CPU: 3
[ 105.215861] func: cpuhp_thread_fun st->state: 187, line: 696, CPU: 3
[ 105.215955] func: cpuhp_thread_fun st->state: 186, line: 696, CPU: 3
[ 105.216053] func: cpuhp_thread_fun st->state: 185, line: 696, CPU: 3
[ 105.216148] func: cpuhp_thread_fun st->state: 184, line: 696, CPU: 3
[ 105.216246] func: cpuhp_thread_fun st->state: 183, line: 696, CPU: 3
[ 105.216348] func: cpuhp_thread_fun st->state: 182, line: 696, CPU: 3
[ 105.216445] func: cpuhp_thread_fun st->state: 181, line: 696, CPU: 3
[ 105.216548] func: cpuhp_thread_fun st->state: 180, line: 696, CPU: 3
[ 105.216648] func: cpuhp_thread_fun st->state: 179, line: 696, CPU: 3
[ 105.216748] func: cpuhp_thread_fun st->state: 178, line: 696, CPU: 3
[ 105.216878] func: cpuhp_thread_fun st->state: 177, line: 696, CPU: 3
[ 105.216977] func: cpuhp_thread_fun st->state: 176, line: 696, CPU: 3
[ 105.217076] func: cpuhp_thread_fun st->state: 175, line: 696, CPU: 3
[ 105.217174] func: cpuhp_thread_fun st->state: 174, line: 696, CPU: 3
[ 105.217270] func: cpuhp_thread_fun st->state: 173, line: 696, CPU: 3
[ 105.217371] func: cpuhp_thread_fun st->state: 172, line: 696, CPU: 3
[ 105.217461] func: cpuhp_thread_fun st->state: 171, line: 696, CPU: 3
[ 105.217701] func: cpuhp_thread_fun st->state: 170, line: 696, CPU: 3
[ 105.217823] func: cpuhp_thread_fun st->state: 169, line: 696, CPU: 3
[ 105.217931] func: cpuhp_thread_fun st->state: 168, line: 696, CPU: 3
[ 105.218033] func: cpuhp_thread_fun st->state: 167, line: 696, CPU: 3
[ 105.218138] func: cpuhp_thread_fun st->state: 166, line: 696, CPU: 3
[ 105.218248] func: cpuhp_thread_fun st->state: 165, line: 696, CPU: 3
[ 105.218371] func: cpuhp_thread_fun st->state: 164, line: 696, CPU: 3
[ 105.218482] func: cpuhp_thread_fun st->state: 163, line: 696, CPU: 3
[ 105.218594] func: cpuhp_thread_fun st->state: 162, line: 696, CPU: 3
[ 105.218707] func: cpuhp_thread_fun st->state: 161, line: 696, CPU: 3
[ 105.218818] func: cpuhp_thread_fun st->state: 160, line: 696, CPU: 3
[ 105.218922] func: cpuhp_thread_fun st->state: 159, line: 696, CPU: 3
[ 105.219029] func: cpuhp_thread_fun st->state: 158, line: 696, CPU: 3
[ 105.219136] func: cpuhp_thread_fun st->state: 157, line: 696, CPU: 3
[ 105.219243] func: cpuhp_thread_fun st->state: 156, line: 696, CPU: 3
[ 105.219352] func: cpuhp_thread_fun st->state: 155, line: 696, CPU: 3
[ 105.219453] func: cpuhp_thread_fun st->state: 154, line: 696, CPU: 3
[ 105.219565] func: cpuhp_thread_fun st->state: 153, line: 696, CPU: 3
[ 105.219670] func: cpuhp_thread_fun st->state: 152, line: 696, CPU: 3
[ 105.219793] func: cpuhp_thread_fun st->state: 151, line: 696, CPU: 3
[ 105.219907] func: cpuhp_thread_fun st->state: 150, line: 696, CPU: 3
[ 105.220019] func: cpuhp_thread_fun st->state: 149, line: 696, CPU: 3
[ 105.220128] func: cpuhp_thread_fun st->state: 148, line: 696, CPU: 3
[ 105.220240] func: cpuhp_thread_fun st->state: 147, line: 696, CPU: 3
[ 105.220351] func: cpuhp_thread_fun st->state: 146, line: 696, CPU: 3
[ 105.220471] func: cpuhp_thread_fun st->state: 145, line: 696, CPU: 3
[ 105.220577] func: cpuhp_thread_fun st->state: 144, line: 696, CPU: 3
[ 105.220693] func: cpuhp_thread_fun st->state: 143, line: 696, CPU: 3
[ 105.220801] func: cpuhp_thread_fun st->state: 142, line: 696, CPU: 3
[ 105.220912] func: cpuhp_thread_fun st->state: 141, line: 696, CPU: 3
[ 105.221046] func: cpuhp_thread_fun st->state: 140, line: 696, CPU: 3
[ 105.221158] func: cpuhp_thread_fun before complete_ap_thread st->state: 139 , line: 744, CPU: 3
[ 105.221318] func: _cpu_down: after cpuhp_kick_ap_work line: 1096, CPU: 0
[ 105.221607] func: _cpu_down: before cpuhp_down_callbacks line: 1117, CPU: 0
[ 105.221713] func: cpuhp_down_callbacks, st->state: 139, line: 1057 CPU: 0
[ 105.221819] func: takedown_cpu line: 969, CPU: 0
[ 105.221950] func: stop_machine_cpuslocked line: 606, CPU: 0
[ 105.222063] func: cpu_stopper_thread line: 502, CPU: 0
[ 105.222067] func: cpu_stopper_thread line: 502, CPU: 3
[ 105.222143] func: multi_cpu_stop line: 196, CPU: 0
[ 105.222233] func: multi_cpu_stop line: 196, CPU: 3
[ 105.222393] func: take_cpu_down line: 935, CPU: 3
[ 105.222481] func: __cpu_disable line: 294, CPU: 3
[ 105.222628] func: cpu_stop_signal_done line: 64, CPU: 0
[ 105.222631] func: cpu_stop_signal_done line: 64, CPU: 3
[ 105.222863] func: cpu_die line: 367, CPU: 3
[ 105.222879] func: cpuhp_complete_idle_dead line: 1023, CPU: 0
[ 105.222937] psci: func: cpu_psci_cpu_die line: 78, CPU: 3
[ 105.223123] psci: func: psci_cpu_off line: 188, CPU: 3
[ 105.223176] func: takedown_cpu: before wait_for_ap_thread line: 1001, CPU: 0
[ 105.223210] psci: func: __invoke_psci_fn_hvc line: 136, CPU: 3
[ 105.223325] func: takedown_cpu: after wait_for_ap_thread line: 1003, CPU: 0
[ 105.223522] func: takedown_cpu: before __cpu_die line: 1011, CPU: 0
[ 105.223616] func: __cpu_die: before cpu_wait_death line: 337, CPU: 0
[ 105.223712] func: __cpu_die: after cpu_wait_death line: 342, CPU: 0
[ 105.223806] CPU3: shutdown
[ 105.223850] func: __cpu_die: before op_cpu_kill line: 351, CPU: 0
[ 105.223938] func: op_cpu_kill line: 325, CPU: 0
[ 105.224006] psci: func: cpu_psci_cpu_kill line: 94, CPU: 0
[ 105.224089] psci: func: psci_affinity_info line: 219, CPU: 0
[ 105.224172] psci: func: __invoke_psci_fn_hvc line: 136, CPU: 0
[ 105.224268] psci: CPU3 killed (polled 0 ms)
[ 105.224336] func: __cpu_die: after op_cpu_kill line: 353, CPU: 0
[ 105.224431] func: takedown_cpu: after __cpu_die line: 1013, CPU: 0
[ 105.224541] func: cpuhp_down_callbacks, st->state: 87, line: 1057 CPU: 0
[ 105.224658] func: cpuhp_down_callbacks, st->state: 86, line: 1057 CPU: 0
[ 105.224761] func: cpuhp_down_callbacks, st->state: 85, line: 1057 CPU: 0
[ 105.224855] func: cpuhp_down_callbacks, st->state: 84, line: 1057 CPU: 0
[ 105.224961] func: cpuhp_down_callbacks, st->state: 83, line: 1057 CPU: 0
[ 105.225061] func: cpuhp_down_callbacks, st->state: 82, line: 1057 CPU: 0
[ 105.225161] func: cpuhp_down_callbacks, st->state: 81, line: 1057 CPU: 0
[ 105.225258] func: cpuhp_down_callbacks, st->state: 80, line: 1057 CPU: 0
[ 105.225359] func: cpuhp_down_callbacks, st->state: 79, line: 1057 CPU: 0
[ 105.225454] func: cpuhp_down_callbacks, st->state: 78, line: 1057 CPU: 0
[ 105.225590] func: cpuhp_down_callbacks, st->state: 77, line: 1057 CPU: 0
[ 105.225695] func: cpuhp_down_callbacks, st->state: 76, line: 1057 CPU: 0
[ 105.225795] func: cpuhp_down_callbacks, st->state: 75, line: 1057 CPU: 0
[ 105.225893] func: cpuhp_down_callbacks, st->state: 74, line: 1057 CPU: 0
[ 105.225992] func: cpuhp_down_callbacks, st->state: 73, line: 1057 CPU: 0
[ 105.226088] func: cpuhp_down_callbacks, st->state: 72, line: 1057 CPU: 0
[ 105.226184] func: cpuhp_down_callbacks, st->state: 71, line: 1057 CPU: 0
[ 105.226281] func: cpuhp_down_callbacks, st->state: 70, line: 1057 CPU: 0
[ 105.226378] func: cpuhp_down_callbacks, st->state: 69, line: 1057 CPU: 0
[ 105.226480] func: cpuhp_down_callbacks, st->state: 68, line: 1057 CPU: 0
[ 105.226576] func: cpuhp_down_callbacks, st->state: 67, line: 1057 CPU: 0
[ 105.226679] func: cpuhp_down_callbacks, st->state: 66, line: 1057 CPU: 0
[ 105.226797] func: cpuhp_down_callbacks, st->state: 65, line: 1057 CPU: 0
[ 105.226893] func: cpuhp_down_callbacks, st->state: 64, line: 1057 CPU: 0
[ 105.227002] func: cpuhp_down_callbacks, st->state: 63, line: 1057 CPU: 0
[ 105.227096] func: cpuhp_down_callbacks, st->state: 62, line: 1057 CPU: 0
[ 105.227191] func: cpuhp_down_callbacks, st->state: 61, line: 1057 CPU: 0
[ 105.227290] func: cpuhp_down_callbacks, st->state: 60, line: 1057 CPU: 0
[ 105.227445] func: cpuhp_down_callbacks, st->state: 59, line: 1057 CPU: 0
[ 105.227579] func: cpuhp_down_callbacks, st->state: 58, line: 1057 CPU: 0
[ 105.227681] func: cpuhp_down_callbacks, st->state: 57, line: 1057 CPU: 0
[ 105.227784] func: cpuhp_down_callbacks, st->state: 56, line: 1057 CPU: 0
[ 105.227879] func: cpuhp_down_callbacks, st->state: 55, line: 1057 CPU: 0
[ 105.228001] func: cpuhp_down_callbacks, st->state: 54, line: 1057 CPU: 0
[ 105.228098] func: cpuhp_down_callbacks, st->state: 53, line: 1057 CPU: 0
[ 105.228195] func: cpuhp_down_callbacks, st->state: 52, line: 1057 CPU: 0
[ 105.228290] func: cpuhp_down_callbacks, st->state: 51, line: 1057 CPU: 0
[ 105.228388] func: cpuhp_down_callbacks, st->state: 50, line: 1057 CPU: 0
[ 105.228482] func: cpuhp_down_callbacks, st->state: 49, line: 1057 CPU: 0
[ 105.228579] func: cpuhp_down_callbacks, st->state: 48, line: 1057 CPU: 0
[ 105.228677] func: cpuhp_down_callbacks, st->state: 47, line: 1057 CPU: 0
[ 105.228777] func: cpuhp_down_callbacks, st->state: 46, line: 1057 CPU: 0
[ 105.228870] func: cpuhp_down_callbacks, st->state: 45, line: 1057 CPU: 0
[ 105.228971] func: cpuhp_down_callbacks, st->state: 44, line: 1057 CPU: 0
[ 105.229067] func: cpuhp_down_callbacks, st->state: 43, line: 1057 CPU: 0
[ 105.229165] func: cpuhp_down_callbacks, st->state: 42, line: 1057 CPU: 0
[ 105.229272] func: cpuhp_down_callbacks, st->state: 41, line: 1057 CPU: 0
[ 105.229370] func: cpuhp_down_callbacks, st->state: 40, line: 1057 CPU: 0
[ 105.229475] func: cpuhp_down_callbacks, st->state: 39, line: 1057 CPU: 0
[ 105.237927] func: cpuhp_down_callbacks, st->state: 38, line: 1057 CPU: 0
[ 105.238038] func: cpuhp_down_callbacks, st->state: 37, line: 1057 CPU: 0
[ 105.238139] func: cpuhp_down_callbacks, st->state: 36, line: 1057 CPU: 0
[ 105.238260] func: cpuhp_down_callbacks, st->state: 35, line: 1057 CPU: 0
[ 105.238361] func: cpuhp_down_callbacks, st->state: 34, line: 1057 CPU: 0
[ 105.238464] func: cpuhp_down_callbacks, st->state: 33, line: 1057 CPU: 0
[ 105.238574] func: cpuhp_down_callbacks, st->state: 32, line: 1057 CPU: 0
[ 105.238671] func: cpuhp_down_callbacks, st->state: 31, line: 1057 CPU: 0
[ 105.238775] func: cpuhp_down_callbacks, st->state: 30, line: 1057 CPU: 0
[ 105.241146] func: cpuhp_down_callbacks, st->state: 29, line: 1057 CPU: 0
[ 105.241410] func: cpuhp_down_callbacks, st->state: 28, line: 1057 CPU: 0
[ 105.241521] func: cpuhp_down_callbacks, st->state: 27, line: 1057 CPU: 0
[ 105.241702] func: cpuhp_down_callbacks, st->state: 26, line: 1057 CPU: 0
[ 105.241812] func: cpuhp_down_callbacks, st->state: 25, line: 1057 CPU: 0
[ 105.241913] func: cpuhp_down_callbacks, st->state: 24, line: 1057 CPU: 0
[ 105.242012] func: cpuhp_down_callbacks, st->state: 23, line: 1057 CPU: 0
[ 105.242119] func: cpuhp_down_callbacks, st->state: 22, line: 1057 CPU: 0
[ 105.242218] func: cpuhp_down_callbacks, st->state: 21, line: 1057 CPU: 0
[ 105.242321] func: cpuhp_down_callbacks, st->state: 20, line: 1057 CPU: 0
[ 105.245616] func: cpuhp_down_callbacks, st->state: 19, line: 1057 CPU: 0
[ 105.245734] func: cpuhp_down_callbacks, st->state: 18, line: 1057 CPU: 0
[ 105.245833] func: cpuhp_down_callbacks, st->state: 17, line: 1057 CPU: 0
[ 105.245938] func: cpuhp_down_callbacks, st->state: 16, line: 1057 CPU: 0
[ 105.246042] func: cpuhp_down_callbacks, st->state: 15, line: 1057 CPU: 0
[ 105.246140] func: cpuhp_down_callbacks, st->state: 14, line: 1057 CPU: 0
[ 105.247745] func: cpuhp_down_callbacks, st->state: 13, line: 1057 CPU: 0
[ 105.247860] func: cpuhp_down_callbacks, st->state: 12, line: 1057 CPU: 0
[ 105.247964] func: cpuhp_down_callbacks, st->state: 11, line: 1057 CPU: 0
[ 105.248260] func: cpuhp_down_callbacks, st->state: 10, line: 1057 CPU: 0
[ 105.248388] func: cpuhp_down_callbacks, st->state: 9, line: 1057 CPU: 0
[ 105.248487] func: cpuhp_down_callbacks, st->state: 8, line: 1057 CPU: 0
[ 105.248582] func: cpuhp_down_callbacks, st->state: 7, line: 1057 CPU: 0
[ 105.248677] func: cpuhp_down_callbacks, st->state: 6, line: 1057 CPU: 0
[ 105.248779] func: cpuhp_down_callbacks, st->state: 5, line: 1057 CPU: 0
[ 105.248875] func: cpuhp_down_callbacks, st->state: 4, line: 1057 CPU: 0
[ 105.248971] func: cpuhp_down_callbacks, st->state: 3, line: 1057 CPU: 0
[ 105.249066] func: cpuhp_down_callbacks, st->state: 2, line: 1057 CPU: 0
[ 105.249163] func: cpuhp_down_callbacks, st->state: 1, line: 1057 CPU: 0
[ 105.249259] func: _cpu_down: after cpuhp_down_callbacks line: 1119, CPU: 0
# 可以看到 CPU_OFF 最开始是由 AP 自己调用状态机中的回调函数，当调用到第 139（CPUHP_BRINGUP_CPU）个状态时，调用 takedown_cpu 回调进行关核，当被 AP 关闭之后 BP 会调用 AP 后面状态的回调，直至 AP 状态等于 0 （CPUHP_OFFLINE）。
References
[1] linux cpu hotplug_cpu offline-CSDN
[2] https://www.cnblogs.com/pengdonglin137/p/11925299.html
[3] https://zhuanlan.zhihu.com/p/545550388
[4] https://blog.csdn.net/chensong_2000/article/details/129442447
[5] https://blog.csdn.net/z20230508/article/details/143889879
[6] https://blog.csdn.net/big2chris/article/details/99588058