k8s中部署nacos - 技术栈

1 部署nfs

bash 复制代码

# 在k8s的主节点上执行
mkdir -p /appdata/download
cd /appdata/download
git clone https://github.com/nacos-group/nacos-k8s.git

将nacos部署到middleware的命名空间中

bash 复制代码

kubectl create namespace middleware
cd /appdata/download/nacos-k8s
# 创建角色
kubectl create -f deploy/nfs/rbac.yaml -n middleware
# 创建 ServiceAccount 和部署 NFS-Client Provisioner
kubectl apply -f deploy/nfs/deployment.yaml -n middleware
# 创建 NFS StorageClass并不隶属于任何特定的命名空间
kubectl apply -f deploy/nfs/class.yaml
# 验证NFS部署成功
kubectl get pod -l app=nfs-client-provisioner -n middleware

注意deployment.yaml中要填写自己的nfs服务器地址，不能使用默认的

2 部署mysql

调整mysql-nfs.yaml文件中nfs的地址路径,且需要再10.101.12.95这个nfs上先创建好目录，否则不成功。

bash 复制代码

cd /appdata/download/nacos-k8s
kubectl apply -f deploy/mysql/mysql-nfs.yaml -n middleware

这个mysql容器的创建不是一下就创建完成的，需要一点时间，估计还有

mysql虽然看不到，但是容器组中是存在的

3 初始化nacos配置信息

https://github.com/alibaba/nacos/blob/master/distribution/conf/mysql-schema.sql

因为上面mysql都看不到，我怎么执行呢？脚本自动创建了。

4 部署nacos

首次还花点时间，因为要下载镜像。

bash 复制代码

# 默认创建的就是集群环境
kubectl apply -f deploy/nacos/nacos-pvc-nfs.yaml -n middleware

大概用了7分钟。

5 常见问题

5.1 nfs重复创建的一些问题

如果中间出现了下面一些错误，重复执行的话，可以按照下面的操作排查问题

rbac重复创建的问题

bash 复制代码

[root@master1 nacos-k8s]# kubectl create -f deploy/nfs/rbac.yaml
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": clusterroles.rbac.authorization.k8s.io "nfs-client-provisioner-runner" already exists
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": clusterrolebindings.rbac.authorization.k8s.io "run-nfs-client-provisioner" already exists
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": roles.rbac.authorization.k8s.io "leader-locking-nfs-client-provisioner" already exists
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": rolebindings.rbac.authorization.k8s.io "leader-locking-nfs-client-provisioner" already exists

因为脚本是幂等的，所以修改脚本后，使用apply而不是create，即可重复执行。

5.2 mysql删除重建

执行kubectl describe pod mysql-72cxs,发现下面存储路径不正确，我已经调整为nacos_prod，看来这个数据也需要删除。

这个时候执行删除pod，

bash 复制代码

kubectl delete pod mysql-72cxs -n middleware

因为pod会自动重建，系统才会按照新的来。再看挂载是ok的了

但还有异常信息，说明nfs挂载目录，没有/data/k8s_storage/nacos/mysql这个路径，删掉重建就可以了

我的数据名，已经改了，但是数据库还是之前的，看来这个也要删除.

5.3 nacos启动UnknownHostException: jmenv.tbsite.net

安装coredns之后，就没有这个问题了

bash 复制代码

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serverMemberManager' defined in URL [jar:file:/home/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-core-2.0.3.jar!/com/alibaba/nacos/core/cluster/ServerMemberManager.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.core.cluster.ServerMemberManager]: Constructor threw exception; nested exception is ErrCode:500, ErrMsg:jmenv.tbsite.net
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:304)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:285)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1338)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:554)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:514)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:319)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1196)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760)
        ... 98 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.core.cluster.ServerMemberManager]: Constructor threw exception; nested exception is ErrCode:500, ErrMsg:jmenv.tbsite.net
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:187)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:300)
        ... 112 common frames omitted
Caused by: com.alibaba.nacos.api.exception.NacosException: java.net.UnknownHostException: jmenv.tbsite.net
        at com.alibaba.nacos.core.cluster.lookup.AddressServerMemberLookup.run(AddressServerMemberLookup.java:150)
        at com.alibaba.nacos.core.cluster.lookup.AddressServerMemberLookup.doStart(AddressServerMemberLookup.java:98)
        at com.alibaba.nacos.core.cluster.AbstractMemberLookup.start(AbstractMemberLookup.java:53)
        at com.alibaba.nacos.core.cluster.ServerMemberManager.initAndStartLookup(ServerMemberManager.java:181)
        at com.alibaba.nacos.core.cluster.ServerMemberManager.init(ServerMemberManager.java:161)
        at com.alibaba.nacos.core.cluster.ServerMemberManager.<init>(ServerMemberManager.java:142)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:175)

5.4 No DataSource set

bash 复制代码

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'asyncNotifyService': Unsatisfied dependency expressed through field 'dumpService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'externalDumpService': Invocation of init method failed; nested exception is ErrCode:500, ErrMsg:Nacos Server did not start because dumpservice bean construction failure :
No DataSource set
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:598)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1402)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:591)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:514)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:319)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1196)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760)
        ... 27 common frames omitted

查看mysql的日志，发现nacos还是连接的nacos_devtest，虽然我已经创建好了nacos_prod

开始我以为是nacos权限问题

bash 复制代码

GRANT ALL PRIVILEGES ON nacos.* TO 'nacos'@'%' IDENTIFIED BY 'nacos';
FLUSH PRIVILEGES;

查看configmap才发现配置有问题。

可以手工更新一下，然后重启nacos就可以了

bash 复制代码

kubectl edit configmap nacos-cm -n middleware

5.5 应用如何访问

bash 复制代码

# 通过这个地址可以访问
nacos-headless.middleware.svc.cluster.local:8848