由于长时间没有启动 VCF 环境，现在在启动 SDDC Manager 组件后，UI 一直处于如下图所示的"初始化"状态。当时第一直觉就认为肯定是 VCF 环境组件的用户密码过期了，之前在管理 VCF 环境中组件的用户密码和密码策略文章中了解过，VCF 环境中组件的用户密码统一由 SDDC Manager 来管理，比如执行轮换、更新以及修复等，而默认情况下，VCF 组件的用户密码具有有效期，因为很长时间没有进行密码手动/自动轮换，而我也没有及时进行处理，所以猜测现在组件的密码应该已经到期了。
最终，在等待很长一段时间后，网页出现如下所示的报错，或者 UI 提示服务器启动失败。
{"message":"Identity Internal Server Error","code":"IDENTITY_INTERNAL_SERVER_ERROR","status":500}
Server failed to start.Fail to init PSC and/or Postgres.Check the SDDC Manager UI Server logs for more details.
如果你尝试登录 VCF 管理域 vCenter Server VAMI 管理界面，并尝试使用 root 用户登录，结果显示不能认证用户。
如果你尝试登录 VCF 管理域 NSX Manager VIP 管理界面，并尝试使用 admin 用户登录，结果显示您的密码已过期。
之前都是在正常情况下，再去管理 VCF 组件用户密码，现在正好趁这个机会，看看当 VCF 组件密码已经出现过期后，该如何进行解决。
根据上面的情况可以大概判断方向，应该就是 VCF 组件的密码发生过期，进而导致了 SDDC Manager 服务无法启动。所以，我可以通过 SSH 并使用 vcf 用户连接到 SDDC Manager，然后使用 SoS 实用程序来直接检查 VCF 组件的密码健康状态，当然，如果还不知道方向，直接进行所有检查（-- health-check）也行。从下面的输出结果可以发现，vCenter Server 组件的用户密码状态无法获取，问题可能出现在该组件上。
vcf@vcf-mgmt01-sddc01 [ ~ ]$ sudo /opt/vmware/sddc-support/sos --password-health --force
Welcome to Supportability and Serviceability(SoS) utility!
Locks info:
[{'status': 'ACTIVE', 'lockId': '363d9097-4938-4872-9fad-d50a05c4360a', 'createdTimeStamp': 1, 'vcfClientContext': {'serviceIdentifier': 'Password Manager', 'pollingInterval': 0}, 'resourceType': 'DEPLOYMENT', 'description': 'Password management operation in progress. Please wait for completion'}]
Found active workflow for Password Manager, Description: Password management operation in progress. Please wait for completion
User passed --force flag to continue SoS operations although notified about running workflows. Please expect failures with SoS operations.
Performing SoS operation for vcf-mgmt01 domain components
Health Check : /var/log/vmware/vcf/sddc-support/healthcheck-2025-03-17-04-06-48-10241
Health Check log : /var/log/vmware/vcf/sddc-support/healthcheck-2025-03-17-04-06-48-10241/sos.log
NOTE : The Health check operation was invoked without --skip-known-host-check, additional identity checks will be included for Connectivity Health, Password Health and Certificate Health Checks because of security reasons.
SDDC Manager : vcf-mgmt01-sddc01.mulab.local
+-------------------------+-----------+
| Stage | Status |
+-------------------------+-----------+
| Bringup | Completed |
| Management Domain State | Completed |
+-------------------------+-----------+
+--------------------+---------------+
| Component | Identity |
+--------------------+---------------+
| SDDC-Manager | 192.168.32.70 |
| Number of Servers | 4 |
+--------------------+---------------+
Password Expiry Status : RED
+-----+-----------------------------------------+---------------------------+-------------------+--------------+-----------------+-----------------------+
| SL# | Component | User | Last Changed Date | Expiry Date | Expires in Days | State |
+-----+-----------------------------------------+---------------------------+-------------------+--------------+-----------------+-----------------------+
| 1 | ESXI : vcf-mgmt01-esxi01.mulab.local | svc-vcf-vcf-mgmt01-esxi01 | Mar 14, 2025 | Never | Never | GREEN |
| | | root | Mar 14, 2025 | Never | Never | GREEN |
| 2 | ESXI : vcf-mgmt01-esxi02.mulab.local | svc-vcf-vcf-mgmt01-esxi02 | Mar 14, 2025 | Never | Never | GREEN |
| | | root | Mar 14, 2025 | Never | Never | GREEN |
| 3 | ESXI : vcf-mgmt01-esxi03.mulab.local | svc-vcf-vcf-mgmt01-esxi03 | Mar 14, 2025 | Never | Never | GREEN |
| | | root | Mar 14, 2025 | Never | Never | GREEN |
| 4 | ESXI : vcf-mgmt01-esxi04.mulab.local | svc-vcf-vcf-mgmt01-esxi04 | Mar 14, 2025 | Never | Never | GREEN |
| | | root | Mar 14, 2025 | Never | Never | GREEN |
| 5 | SDDC : vcf-mgmt01-sddc01.mulab.local | vcf | Mar 17, 2025 | Apr 21, 2026 | 400 days | GREEN |
| | | backup | Dec 07, 2024 | Jan 11, 2026 | 300 days | GREEN |
| | | root | Dec 07, 2024 | Jan 11, 2026 | 300 days | GREEN |
| 6 | vCenter : vcf-mgmt01-vcsa01.mulab.local | root | - | - | - | Failed to get details |
+-----+-----------------------------------------+---------------------------+-------------------+--------------+-----------------+-----------------------+
Legend:
GREEN - No attention required, health status is NORMAL
YELLOW - May require attention, health status is WARNING
RED - Requires immediate attention, health status is CRITICAL
Health Check completed successfully for : [VCF-SUMMARY]
Operation failed for : [PASSWORD-CHECK(1/12 Failed)]
For detailed report please refer : /var/log/vmware/vcf/sddc-support/healthcheck-2025-03-17-04-06-48-10241/report.json
vcf@vcf-mgmt01-sddc01 [ ~ ]$通过在 SDDC Manager 上查阅 /var/log/vmware/vcf/domainmanager/domainmanager.log 日志，找到以下信息。判断大概是使用了不正确的用户名或密码，导致连接 vCenter Server 失败。
2025-03-17T03:54:54.221+0000 ERROR [vcf_dm,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VsphereClient,ForkJoinPool.commonPool-worker-4] Failed to connect to https://vcf-mgmt01-vcsa01.mulab.local:443/sdk
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.connect(VcManagerBase.java:514)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:495)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:468)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.getVcManagerBase(VcManagerFactory.java:436)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:52)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:157)
at com.vmware.vcf.configreconciler.config.checks.ClusterHaSettingsConfigDriftCheck.isConfigurationRealized(ClusterHaSettingsConfigDriftCheck.java:44)
at com.vmware.vcf.configreconciler.service.ResourceCacheService.performIsConfigRealizedCheck(ResourceCacheService.java:1014)
at com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshFailedConfigRealizedCheck(ResourceCacheService.java:1055)
at com.vmware.vcf.configreconciler.service.ResourceCacheService.refreshConfigApplicability(ResourceCacheService.java:935)
at com.vmware.vcf.configreconciler.service.ResourceCacheService.lambda$refreshLocalInventoryWithConfigApplicability$0(ResourceCacheService.java:264)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
at java.base/java.lang.Class.newInstance(Class.java:645)
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:167)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:105)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:92)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:86)
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:288)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:243)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:267)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:56)
at com.vmware.vim.vmomi.core.tracing.NoopTracer$NoopSpan.runWithinSpanContext(NoopTracer.java:120)
at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:17)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:54)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:229)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:119)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
at jdk.proxy2/jdk.proxy2.$Proxy287.login(Unknown Source)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:120)
... 21 common frames omitted
2025-03-17T03:54:54.235+0000 WARN [vcf_dm,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VsphereClient,ForkJoinPool.commonPool-worker-4] Error logging out of session
com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.通过在 SDDC Manager 上查阅 /var/log/vmware/vcf/operationsmanager/operationsmanager.log 日志，找到以下信息。从这里可以获取到更精确的信息，判断大概是 [email protected] 用户的凭据不正确，导致无法连接到 vCenter Server（[email protected]）。
2025-03-17T08:52:47.868+0000 ERROR [vcf_om,a20e030c092d465d,8a4d] [c.v.e.s.c.c.v.vsphere.VcManagerBase,http-nio-127.0.0.1-7300-exec-1] Cannot complete login due to incorrect credentials: vcf-mgmt01-vcsa01.mulab.local, [email protected].
2025-03-17T08:52:47.875+0000 DEBUG [vcf_om,a20e030c092d465d,8a4d] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7300-exec-1] Fetching Management vCenter data from inventory
2025-03-17T08:52:47.875+0000 DEBUG [vcf_om,a20e030c092d465d,8a4d] [c.v.e.s.c.s.a.i.InventoryServiceAdapterImpl,http-nio-127.0.0.1-7300-exec-1] Fetching vCenters data from inventory
2025-03-17T08:52:47.897+0000 ERROR [vcf_om,a20e030c092d465d,8a4d] [c.v.v.r.s.PluginInfoServiceImpl,http-nio-127.0.0.1-7300-exec-1] Exception when calling get extension: com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot connect to vCenter Server vcf-mgmt01-vcsa01.mulab.local
at com.vmware.vcf.remoteplugin.service.PluginRegistrationServiceImpl.get(PluginRegistrationServiceImpl.java:174)
at com.vmware.vcf.remoteplugin.service.PluginInfoServiceImpl.initSddcManagerPlugin(PluginInfoServiceImpl.java:75)
at com.vmware.vcf.remoteplugin.controller.PluginRegistrationController.initSddcManagerPlugin(PluginRegistrationController.java:53)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)如果在 SDDC Manager 上查阅 /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log 日志，发现也有一样的信息。
2025-03-17T03:43:16.119+0000 ERROR [vcf_om,0000000000000000,0000] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-scheduler-2] Cannot complete login due to incorrect credentials: vcf-mgmt01-vcsa01.mulab.local, [email protected].
2025-03-17T03:43:16.119+0000 WARN [vcf_om,0000000000000000,0000] [c.v.v.t.services.TaskPublisher,om-scheduler-2] Failed to check for leftover tasks
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.connect(VcManagerBase.java:514)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:495)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:468)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.getVcManagerBase(VcManagerFactory.java:436)
at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:52)
at com.vmware.vcf.taskpublisher.services.TaskPublisher.checkForLeftoverTasks(TaskPublisher.java:343)
at com.vmware.vcf.taskpublisher.services.TaskPublisher.cloneTasks(TaskPublisher.java:102)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)如果你登录 vSphere Client，在 vCenter Server"事件控制台"中，可能也会发现同上述日志一样的错误信息。
所有的信息都指向那个 SVC 用户，这到底是一个什么用户？在 VCF 环境它被称为服务账户（Service Account），服务账户类型（SERVICE）由 VMware Cloud Foundation 自动创建，用于产品组件之间的交互。服务账户（Service Account）通常不需要我们手动干预，但是一些特殊情况除外。
综合以上信息，可以大概判断是 SVC 用户的凭据不正确导致无法连接到 SDDC Manager 或者 SDDC Manager 无法连接到 vCenter Server 进行认证，这种情况有可能是 SDDC Manager 自动轮换了 SVC 用户的密码，但是只更新了自身的数据库，vCenter Server 由于长期没有启动并且用户密码已经过期，所以导致无法进行连接 vCenter Server 并对 SVC 密码进行更新。
现在有两个办法，一是在 vCenter Server 上找到这个 SVC 用户的密码，然后去 SDDC Manager 更新这个 SVC 用户的密码；另一个就是反过来。很显然，第一种方法不太可能找到，但可幸的是官方提供了第二种方法。
参考 KB 327195 知识库，可以通过 SDDC Manager 获取服务账户（Service Account）的凭据密码，通过 SSH 以 vcf 用户登录到 SDDC Manager 并使用"su - "切换到 root 用户，然后使用以下命令获取到 SDDC Manager API 的连接 token 凭据。
TOKEN=$(curl -d '{"username" : [email protected], "password" : "Vcf521@password"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')然后再运行以下命令获取所有服务账户的凭据密码。
curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less
通过运行以上命令，你应该可以找到对应服务账户的凭据密码，"username"为服务账户，"secret"为凭据密码。
然后登录到 vCenter Server（vSphere Client），在系统管理中找到用户管理，在用户中选择 SSO 域（vsphere.local），找到服务账户并点击"编辑"，将密码修改为上面获取到的密码。
通过 SSH 以 vcf 用户登录到 SDDC Manager 并使用"su - "切换到 root 用户，运行以下命令重启所有服务，或者也可以直接重启 SDDC Manager 电源。
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
如果一切顺利，应该能正常登录到 SDDC Manager UI。导航到安全->密码管理，可以发现 vCenter Server 和 NSX Manager 组件的用户因为密码过期都已经从 SDDC Manager 断开连接。
vCenter Server 组件的账户可以直接使用密码轮换（Rotate Password）功能，这样可以直接更新并重新进行连接。
NSX Manager 组件的账户不能使用密码轮换，只能手动更改账户的密码后再使用密码修复（Remediate Password）功能。手动修改 NSX Manager 账户的密码支持多种方法（GUI、CLI、API 等），比如可以参考 KB 314657 知识库中的方法，使用 CLI 方式进行修改。
如果不知道 NSX Manager 组件账户的密码，可以在 SDDC Manager 上通过 lookup_passwords 命令查看密码。
通过 SSH 并使用 root 用户登录到 NSX Manager CLI 管理界面，然后依次执行以下操作即可。注意，如果 NSX Manager 还未启动 SSH 服务，可以通过 vCenter Server 打开 NSX Manager 虚拟机的控制台，然后输入管理员用户和密码登录，运行"start service ssh"启动服务，运行"set service ssh start-on-boot"设置开机自启动，运行"set ssh root-login"允许 root 登录，运行"get service ssh"获取服务状态。
echo "" >/etc/security/opasswd
/etc/init.d/nsx-mp-api-server stop
passwd
passwd admin
passwd audit
touch /var/vmware/nsx/reset_cluster_credentials
/etc/init.d/nsx-mp-api-server start
/usr/sbin/faillock --user root --reset
/usr/sbin/faillock --user admin --reset
/usr/sbin/faillock --user audit --reset完成 NSX Manager 账户密码修改后，回到 SDDC Manager 密码管理视图，点击用户旁边的"修复密码"，输入修改后的新密码即可。
依次完成所有账户的修复，现在一切恢复正常。
