聊一聊 操作系统蓝屏 c0000102 的故障分析

一:背景

1. 讲故事

今年以来不知道为啥总有些朋友加我微信,让我帮忙分析下操作系统蓝屏问题,我也觉得挺好奇的,就问了其中一位朋友,说是B站来的,我就在拼命回忆,为啥会找我分析蓝屏?突然想到了去年好像录了一集关于 CrowdStrike 的热点事件。。。我本来是做.NET程序的故障分析,这操作系统的蓝屏虽然不是我擅长的,但也能硬着头皮上吧。

这位朋友找到我的时候,说它已经重装过操作系统,但还是会偶发性蓝屏,让我看下到底是啥原因导致的,那就上 windbg 分析吧。

二:为什么会蓝屏

1. 到底崩溃在哪里

要想找到崩溃点,其实 windbg 提供了一个牛叉的自动化分析命令!analyze -v,它可以帮我们挖出崩溃上下文,输出如下:

C# 复制代码
14: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Unknown bugcheck code (c0000102)
Unknown bugcheck description
Arguments:
Arg1: ffffab027a2c6030
Arg2: ffffab02785d06a0
Arg3: 0000000000000000
Arg4: 0000000000000000

PROCESS_NAME:  csrss.exe

STACK_TEXT:  
ffff930a`26342578 fffff800`ef921e61     : 00000000`0000004c 00000000`c0000102 ffff930a`2733f390 ffff9703`d52fc310 : nt!KeBugCheckEx
ffff930a`26342580 fffff800`ef93266d     : ffffffff`ffffffff ffff930a`26342699 ffffffff`80000a04 00000000`00000000 : nt!PopGracefulShutdown+0x2d9
ffff930a`263425e0 fffff800`ef9101e0     : 00000000`00000001 fffff800`00000000 00000000`00000000 00000000`00000000 : nt!PopTransitionSystemPowerStateEx+0x895
ffff930a`26342700 fffff800`ef8632bd     : 00000000`00000047 00000000`ffffff00 00000000`00000001 00000000`00000000 : nt!PopTransitionSystemPowerState+0x4c
ffff930a`263428b0 fffff800`ef862e80     : 00000000`00000000 ffff9703`00000005 00000000`00000001 00000000`c0000004 : nt!PopIssueActionRequest+0x2a9
ffff930a`26342950 fffff800`ef255d3e     : 00000000`00000001 00000000`00000000 00000000`fffffffb ffff9703`c5620aa0 : nt!PopPolicyWorkerAction+0x80
ffff930a`263429d0 fffff800`ef1704a2     : ffff9703`00000000 ffff9703`c57d4040 ffff930a`26342b00 ffff9703`c5620aa0 : nt!PopPolicyWorkerThread+0x7e
ffff930a`26342a00 fffff800`ef25585a     : ffff9703`c57d4040 ffff9703`c57d4040 fffff800`ef1702f0 ffff9703`c5620aa0 : nt!ExpWorkerThread+0x1b2
ffff930a`26342bb0 fffff800`ef47ae54     : ffffd901`dbd90180 ffff9703`c57d4040 fffff800`ef255800 00000000`00000000 : nt!PspSystemThreadStartup+0x5a
ffff930a`26342c00 00000000`00000000     : ffff930a`26343000 ffff930a`2633c000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x34


SYMBOL_NAME:  nt!PopTransitionSystemPowerStateEx+895

MODULE_NAME: nt

IMAGE_NAME:  ntkrnlmp.exe

IMAGE_VERSION:  10.0.26100.2605
...

从卦中可以看到两点信息:

  1. bugcode=c0000102

微软定义的蓝屏数字都比较小,所以看到这个码也挺奇怪的,从微软的msdn一查原来是NTSTATUS值,它表示 STATUS_FILE_CORRUPT_ERROR 即文件损坏。参见:https://learn.microsoft.com/zh-cn/troubleshoot/azure/virtual-machines/windows/error-code-0xc0000102-status-file-corrupt

  1. PopGracefulShutdown

从函数的前缀 Pop 可以看出,它是属于内核 电源管理执行体,翻译过来应该叫 Power on Plug,正在做系统的关机操作,当然这里的关机有可能是朋友做了 Close 或者 Restart 操作。

为了能够知己知彼,得需要观察下 PopGracefulShutdown 方法都在干嘛?

2. PopGracefulShutdown 在干嘛

要想找到这个答案,可以在 Reactos 中寻找答案,参考代码如下:

C# 复制代码
VOID
NTAPI
PopGracefulShutdown(IN PVOID Context)
{
    PEPROCESS Process = NULL;

    /* Process the registered waits and work items */
    PopProcessShutDownLists();

    /* Loop every process */
    Process = PsGetNextProcess(Process);
    while (Process)
    {
        /* Make sure this isn't the idle or initial process */
        if ((Process != PsInitialSystemProcess) && (Process != PsIdleProcess))
        {
            /* Print it */
            DPRINT1("%15s is still RUNNING (%p)\n", Process->ImageFileName, Process->UniqueProcessId);
        }

        /* Get the next process */
        Process = PsGetNextProcess(Process);
    }

    /* First, the HAL handles any "end of boot" special functionality */
    DPRINT("HAL shutting down\n");
    HalEndOfBoot();

    /* Shut down the Shim cache if enabled */
    ApphelpCacheShutdown();

    /* In this step, the I/O manager does first-chance shutdown notification */
    DPRINT("I/O manager shutting down in phase 0\n");
    IoShutdownSystem(0);

    /* In this step, all workers are killed and hives are flushed */
    DPRINT("Configuration Manager shutting down\n");
    CmShutdownSystem();

    /* Shut down the Executive */
    DPRINT("Executive shutting down\n");
    ExShutdownSystem();

    /* Note that modified pages should be written here (MiShutdownSystem) */
    MmShutdownSystem(0);

    /* Flush all user files before we start shutting down IO */
    /* This is where modified pages are written back by the IO manager */
    CcShutdownSystem();

    /* In this step, the I/O manager does last-chance shutdown notification */
    DPRINT("I/O manager shutting down in phase 1\n");
    IoShutdownSystem(1);
    CcWaitForCurrentLazyWriterActivity();

    /* FIXME: Calling Mm shutdown phase 1 here to get page file dereference
     * but it shouldn't be called here. Only phase 2 should be called.
     */
    MmShutdownSystem(1);

    /* Note that here, we should broadcast the power IRP to devices */

    /* In this step, the HAL disables any wake timers */
    DPRINT("Disabling wake timers\n");
    HalSetWakeEnable(FALSE);

    /* And finally the power request is sent */
    DPRINT("Taking the system down\n");
    PopShutdownSystem(PopAction.Action);
}

从卦中的源代码看,这个关机操作主要是用来关闭各种执行体,那到底是哪一句导致的 KeBugCheckEx 呢?这就需要从汇编上寻找答案,使用 ub 命令即可。

C# 复制代码
14: kd> ub fffff800`ef921e61
nt!PopGracefulShutdown+0x2b5:
fffff800`ef921e3d 488b0dbc843e00  mov     rcx,qword ptr [nt!PopAction+0x40 (fffff800`efd0a300)]
fffff800`ef921e44 488b4140        mov     rax,qword ptr [rcx+40h]
fffff800`ef921e48 4c8b4938        mov     r9,qword ptr [rcx+38h]
fffff800`ef921e4c 4c8b4130        mov     r8,qword ptr [rcx+30h]
fffff800`ef921e50 488b5128        mov     rdx,qword ptr [rcx+28h]
fffff800`ef921e54 8b4920          mov     ecx,dword ptr [rcx+20h]
fffff800`ef921e57 4889442420      mov     qword ptr [rsp+20h],rax
fffff800`ef921e5c e86f6799ff      call    nt!KeBugCheckEx (fffff800`ef2b85d0)

从汇编中可以看到 KeBugCheckEx 中的各种值是来自于 PopAction 结构体,这个结构体是关机操作的一个上下文类 _POP_POWER_ACTION,接下来根据汇编代码用 dt 给导出来,参考如下:

C# 复制代码
14: kd> dt nt!_POP_POWER_ACTION fffff800`efd0a300-0x40
   +0x000 Updates          : 0 ''
   +0x001 State            : 0x3 ''
   +0x002 Shutdown         : 0x1 ''
   +0x004 Action           : 5 ( PowerActionShutdownReset )
   +0x008 LightestState    : 6 ( PowerSystemShutdown )
   +0x00c Flags            : 0xc0000004
   +0x010 Status           : 0n0
   +0x014 DeviceType       : 4 ( PolicyInitiatePowerActionAPI )
   +0x018 DeviceTypeFlags  : 0
   +0x020 RequestorInformation : 0xffffab02`785e2b50 _DIAGNOSTIC_BUFFER
   +0x028 IrpMinor         : 0x2 ''
   +0x029 Waking           : 0 ''
   +0x02c SystemState      : 6 ( PowerSystemShutdown )
   +0x030 NextSystemState  : 1 ( PowerSystemWorking )
   +0x034 EffectiveSystemState : 6 ( PowerSystemShutdown )
   +0x038 CurrentSystemState : 1 ( PowerSystemWorking )
   +0x040 ShutdownBugCode  : 0xffff930a`2733f250 _POP_SHUTDOWN_BUG_CHECK
   +0x048 DevState         : 0xffff9703`d2bf71d0 _POP_DEVICE_SYS_STATE
   +0x050 HiberContext     : (null) 
   +0x058 WakeTime         : 0
   +0x060 SleepTime        : 0
   +0x068 WakeFirstUnattendedTime : 0
   +0x070 WakeAlarmSignaled : 0 ( PoAc )
   +0x078 WakeAlarm        : [3] <unnamed-tag>
   +0x0c0 WakeAlarmPaused  : 0 ''
   +0x0c8 WakeAlarmLastTime : 0
   +0x0d0 DozeDeferralStartTime : 0
   +0x0d8 FilteredCapabilities : SYSTEM_POWER_CAPABILITIES
   +0x128 WatchdogLock     : 0
   +0x130 WatchdogDpc      : _KDPC
   +0x170 WatchdogTimer    : _KTIMER
   +0x1b0 WatchdogInitialized : 0x1 ''
   +0x1b4 WatchdogState    : 0 ( PopPowerActionWatchdogStateDisabled )
   +0x1b8 WatchdogStartTime : 0
   +0x1c0 WatchdogTimeout  : 0
   +0x1c8 ActionWorkerThread : 0xffff9703`c57d4040 _KTHREAD
   +0x1d0 PromoteActionWorkerThread : (null) 
   +0x1d8 UnlockAfterSleepWorkerThread : (null) 

14: kd> dx -id 0,0,ffff9703d27541c0 -r1 ((ntkrnlmp!_POP_SHUTDOWN_BUG_CHECK *)0xffff930a2733f250)
((ntkrnlmp!_POP_SHUTDOWN_BUG_CHECK *)0xffff930a2733f250)                 : 0xffff930a2733f250 [Type: _POP_SHUTDOWN_BUG_CHECK *]
    [+0x000] InitiatingThread : 0xffff9703d274f140 [Type: _ETHREAD *]
    [+0x008] InitiatingProcess : 0xffff9703d27541c0 [Type: _EPROCESS *]
    [+0x010] ThreadId         : 0x4dc [Type: void *]
    [+0x018] ProcessId        : 0x4d8 [Type: void *]
    [+0x020] Code             : 0x4c [Type: unsigned long]
    [+0x028] Parameter1       : 0xc0000102 [Type: unsigned __int64]
    [+0x030] Parameter2       : 0xffff930a2733f390 [Type: unsigned __int64]
    [+0x038] Parameter3       : 0xffff9703d52fc310 [Type: unsigned __int64]
    [+0x040] Parameter4       : 0xffff9703d53bdcc0 [Type: unsigned __int64]

从卦中可以看到如下信息:

  1. Action=5

原来这是一个 PowerActionShutdownReset 操作,即 重启 操作。

  1. Code=0x4c

从 _POP_SHUTDOWN_BUG_CHECK 结构体中可以看到,真正的 BugCheck=0x4c,而导致 0x4c 的真正原因是 0xc0000102,这个在 !anlayze -v 中没有体现出来。

顺带提醒一下,也可以用 kb 2Args to Child 区域直接观察父子异常码。

C# 复制代码
14: kd> kb 2
 # RetAddr               : Args to Child                                                           : Call Site
00 fffff800`ef921e61     : 00000000`0000004c 00000000`c0000102 ffff930a`2733f390 ffff9703`d52fc310 : nt!KeBugCheckEx
01 fffff800`ef93266d     : ffffffff`ffffffff ffff930a`26342699 ffffffff`80000a04 00000000`00000000 : nt!PopGracefulShutdown+0x2d9

3. 到底是什么文件损坏了

在 reactos 中没有找到对 _POP_SHUTDOWN_BUG_CHECK 结构的相关赋值逻辑,所以此时只能在几个 Parameter 参数上下手,可以 du 观察这几个参数。

C# 复制代码
14: kd> db 0xffff930a2733f390
ffff930a`2733f390  30 60 2c 7a 02 ab ff ff-a0 06 5d 78 02 ab ff ff  0`,z......]x....
ffff930a`2733f3a0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
ffff930a`2733f3b0  00 00 00 00 00 00 00 00-20 25 73 20 25 73 0a 00  ........ %s %s..
ffff930a`2733f3c0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
ffff930a`2733f3d0  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
ffff930a`2733f3e0  54 68 65 20 66 69 6c 65-20 6f 72 20 64 69 72 65  The file or dire
ffff930a`2733f3f0  63 74 6f 72 79 20 73 78-73 73 72 76 20 69 73 20  ctory sxssrv is 
ffff930a`2733f400  63 6f 72 72 75 70 74 20-61 6e 64 20 75 6e 72 65  corrupt and unre

14: kd> db 0xffff9703d52fc310
ffff9703`d52fc310  0a 53 54 4f 50 3a 20 63-30 30 30 30 31 30 32 20  .STOP: c0000102 
ffff9703`d52fc320  7b 43 6f 72 72 75 70 74-20 46 69 6c 65 7d 0a 00  {Corrupt File}..
ffff9703`d52fc330  a6 01 03 02 44 50 44 69-71 d0 64 bc 00 00 00 00  ....DPDiq.d.....
ffff9703`d52fc340  5c 00 44 00 72 00 69 00-76 00 65 00 72 00 5c 00  \.D.r.i.v.e.r.\.
ffff9703`d52fc350  41 00 6d 00 64 00 50 00-50 00 4d 00 00 00 00 00  A.m.d.P.P.M.....
ffff9703`d52fc360  b7 01 03 02 44 50 44 69-3d d5 a1 9c 00 00 00 00  ....DPDi=.......
ffff9703`d52fc370  5c 00 44 00 72 00 69 00-76 00 65 00 72 00 5c 00  \.D.r.i.v.e.r.\.
ffff9703`d52fc380  61 00 6d 00 64 00 69 00-32 00 63 00 00 00 00 00  a.m.d.i.2.c.....


14: kd> da /c100 0xffff9703d53bdcc0
ffff9703`d53bdcc0  "The file or directory sxssrv is corrupt and unreadable...Please run the Chkdsk utility..."

Parameter4 上可以看到原来是 sxssrv.dll 损坏了,通过网络查了下是个 windows子系统文件,所以这个是罪魁祸首,截图如下:

到这里我们大概就捋出来了,原来是客户在电脑重启的过程中,操作系统发现 sxssrv.dll 文件有损,而它又是操作系统非常重要的一个系统文件,所以就蓝屏了。

4. 如何解决

  1. 使用 sfc /scannow 修复受保护的系统文件,并自动修复。
  2. 使用 DISM /Online /Cleanup-Image /RestoreHealth 修复系统镜像。

如果上面都搞不定,并且重装系统也不行,那大概率就是内存或者硬盘的问题了。

  1. 使用 diskgenius 工具的 坏道检测和修复功能,观察硬盘的健康状态。
  2. 使用内置的 Windows内存诊断 工具诊断内存健康度。

如果有问题,那该换的就换。

三:总结

操作系统的代码固若金汤,我们常人基本上遇不到代码导致的bug,更多的是第三方配件导致的不给力,解决此类问题大多都是修复,更换。。。