MongoDB分片集群搭建

Loganer2024-11-23 20:50
  • 什么是分片
    分片是一种跨多台机器分布数据的方法。MongoDB 使用分片来支持超大数据集和高吞吐量操作的部署。
  • 什么情况下使用分片
  1. 存储容量受单机限制，即磁盘资源遭遇瓶颈。
  2. 读写能力受单机限制，可能是CPU、内存或者网卡等资源遭遇瓶颈，导致读写能力无法扩展。
  • Mongo分片组件

    路由节点(Router) 分片集群的访问入口，其本身并不持久化数据。
    配置节点(Config) 配置服务器包含多个节点，并组成一个复制集结构。
    数据节点(Shard) 分片用于存储真正的数据，并提供最终的数据读写访问。
  • Mongo分片方式
  1. 基于范围
  2. 基于哈希
  3. 自定义Zone（区域/范围）
  1. 每台机器安装同版本的Mongo
shell 复制代码 
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-4.4.6.tgz
tar xzvf mongodb-linux-x86_64-rhel70-4.4.6.tgz -C /usr/local
cd /usr/local/
ln -s /usr/local/mongodb-linux-x86_64-rhel70-4.4.6 /usr/local/mongodb
vim /etc/profile

export PATH=/usr/local/mongodb/bin/:$PATH

source /etc/profile
  1. shard1主机操作
    准备存放数据和日志的目录
shell 复制代码 
mkdir -p /usr/local/mongodb/sharded_cluster/myshardrs01_27018/log \
/usr/local/mongodb/sharded_cluster/myshardrs01_27018/data/db \
/usr/local/mongodb/sharded_cluster/myshardrs01_27118/log \
/usr/local/mongodb/sharded_cluster/myshardrs01_27118/data/db \
/usr/local/mongodb/sharded_cluster/myshardrs01_27218/log \
/usr/local/mongodb/sharded_cluster/myshardrs01_27218/data/db

创建配置文件

myshardrs01_27018

记得替换对应的bindIp

shell 复制代码 
vim /usr/local/mongodb/sharded_cluster/myshardrs01_27018/mongod.conf

systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/myshardrs01_27018/log/mongod.log"
    logAppend: true
storage:
    dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27018/data/db"
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27018/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.10
    port: 27018
replication:
    replSetName: myshardrs01
sharding:
  clusterRole: shardsvr

myshardrs01_27118

记得替换对应的bindIp

shell 复制代码 
vim /usr/local/mongodb/sharded_cluster/myshardrs01_27118/mongod.conf

systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/myshardrs01_27118/log/mongod.log"
    logAppend: true
storage:
    dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27118/data/db"
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27118/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.10
    port: 27118
replication:
    replSetName: myshardrs01
sharding:
  clusterRole: shardsvr

myshardrs01_27218

记得替换对应的bindIp

shell 复制代码 
vim /usr/local/mongodb/sharded_cluster/myshardrs01_27218/mongod.conf

systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/myshardrs01_27218/log/mongod.log"
    logAppend: true
storage:
    dbPath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27218/data/db"
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/myshardrs01_27218/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.10
    port: 27218
replication:
    replSetName: myshardrs01
sharding:
  clusterRole: shardsvr

启动第一套副本集：一主一副本一仲裁

shell 复制代码 
/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27018/mongod.conf
/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27118/mongod.conf
/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myshardrs01_27218/mongod.conf

验证一下进程是否正常
ps -ef|grep mongod

连接主节点

shell 复制代码 
mongo --host ip.ip.ip.ip --port 27018
shell 复制代码 
rs.initiate()
{
        "info2" : "no configuration specified. Using a default configuration for the set",
        "me" : "192.168.112.10:27018",
        "ok" : 1
}
myshardrs01:SECONDARY>
myshardrs01:PRIMARY>

添加副节点和仲裁节点

shell 复制代码 
rs.add("192.168.112.10:27118")
{
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1714658449, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1714658449, 1)
}
shell 复制代码 
rs.addArb("192.168.112.10:27218")
{
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1714658509, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1714658509, 1)
}

查看副本集
rs.conf()
rs.status()

  1. shard2主机操作
    类似shard1
  2. 配置节点副本集的创建
    准备存放数据和日志的目录
shell 复制代码 
mkdir -p /usr/local/mongodb/sharded_cluster/myconfigrs_27019/log \
/usr/local/mongodb/sharded_cluster/myconfigrs_27019/data/db \
/usr/local/mongodb/sharded_cluster/myconfigrs_27119/log \
/usr/local/mongodb/sharded_cluster/myconfigrs_27119/data/db \
/usr/local/mongodb/sharded_cluster/myconfigrs_27219/log \
/usr/local/mongodb/sharded_cluster/myconfigrs_27219/data/db

创建配置文件

myconfigrs_27019

shell 复制代码 
systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/myconfigrs_27019/log/mongod.log"
    logAppend: true
storage:
    dbPath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27019/data/db"
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27019/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.30
    port: 27019
replication:
    replSetName: myconfigr
sharding:
    clusterRole: configsvr

myconfigrs_27119

shell 复制代码 
systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/myconfigrs_27119/log/mongod.log"
    logAppend: true
storage:
    dbPath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27119/data/db"
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27119/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.30
    port: 27119
replication:
    replSetName: myconfigr
sharding:
    clusterRole: configsvr

myconfigrs_27219

shell 复制代码 
systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/myconfigrs_27219/log/mongod.log"
    logAppend: true
storage:
    dbPath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27219/data/db"
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/myconfigrs_27219/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.30
    port: 27219
replication:
    replSetName: myconfigr
sharding:
    clusterRole: configsvr

启动副本集

shell 复制代码 
/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myconfigrs_27019/mongod.conf
/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myconfigrs_27119/mongod.conf
/usr/local/mongodb/bin/mongod -f /usr/local/mongodb/sharded_cluster/myconfigrs_27219/mongod.conf

ps检查进程是否正常

连接主节点
mongo --host 192.168.112.30 --port 27019

初始化副本集

shell 复制代码 
rs.initiate()
{
        "info2" : "no configuration specified. Using a default configuration for the set",
        "me" : "192.168.112.30:27019",
        "ok" : 1,
        "$gleStats" : {
                "lastOpTime" : Timestamp(1714702204, 1),
                "electionId" : ObjectId("000000000000000000000000")
        },
        "lastCommittedOpTime" : Timestamp(0, 0)
}

添加副本节点

shell 复制代码 
rs.add("192.168.112.30:27119")
{
        "ok" : 1,
        "$gleStats" : {
                "lastOpTime" : {
                        "ts" : Timestamp(1714702333, 2),
                        "t" : NumberLong(1)
                },
                "electionId" : ObjectId("7fffffff0000000000000001")
        },
        "lastCommittedOpTime" : Timestamp(1714702333, 2),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1714702335, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1714702333, 2)
}

添加副本节点

shell 复制代码 
rs.add("192.168.112.30:27219")
{
        "ok" : 1,
        "$gleStats" : {
                "lastOpTime" : {
                        "ts" : Timestamp(1714702365, 2),
                        "t" : NumberLong(1)
                },
                "electionId" : ObjectId("7fffffff0000000000000001")
        },
        "lastCommittedOpTime" : Timestamp(1714702367, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1714702367, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1714702365, 2)
}
  1. 路由节点的创建和操作
shell 复制代码 
mkdir -p /usr/local/mongodb/sharded_cluster/mymongos_27017/log

创建配置文件

shell 复制代码 
vim /usr/local/mongodb/sharded_cluster/mymongos_27017/mongos.conf

systemLog:
    destination: file
    path: "/usr/local/mongodb/sharded_cluster/mymongos_27017/log/mongod.log"
    logAppend: true
processManagement:
    fork: true
    pidFilePath: "/usr/local/mongodb/sharded_cluster/mymongos_27017/log/mongod.pid"
net:
    bindIp: localhost,192.168.112.40
    port: 27017
sharding:
    configDB: myconfigrs/192.168.112.30:27019,192.168.112.30:27119,192.168.112.30:27219

启动
/usr/local/mongodb/bin/mongos -f /usr/local/mongodb/sharded_cluster/mymongos_27017/mongos.conf

客户端登录mongo
mongo --port 27017

此时写入不了数据

通过路由节点操作，现在只是连接了配置节点

还没有连接分片数据节点，因此无法写入业务数据

  1. 添加分片副本集

第一套分片加进来

shell 复制代码 
sh.addShard("myshardrs01/192.168.112.10:27018,192.168.112.10:27118,192.168.112.10:27218")
{
        "shardAdded" : "myshardrs01",
        "ok" : 1,
        "operationTime" : Timestamp(1714705107, 3),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1714705107, 3),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

查看分片状态

shell 复制代码 
sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("6634477c7677719ed2388706")
  }
  shards:
        {  "_id" : "myshardrs01",  "host" : "myshardrs01/192.168.112.10:27018,192.168.112.10:27118",  "state" : 1 }
  most recently active mongoses:
        "4.4.6" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }

第二套分片加进来

shell 复制代码 
sh.addShard("myshardrs02/192.168.112.20:27318,192.168.112.20:27418,192.168.112.20:27518")
{
        "shardAdded" : "myshardrs02",
        "ok" : 1,
        "operationTime" : Timestamp(1714705261, 2),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1714705261, 2),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

查看分片状态

shell 复制代码 
sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("6634477c7677719ed2388706")
  }
  shards:
        {  "_id" : "myshardrs01",  "host" : "myshardrs01/192.168.112.10:27018,192.168.112.10:27118",  "state" : 1 }
        {  "_id" : "myshardrs02",  "host" : "myshardrs02/192.168.112.20:27318,192.168.112.20:27418",  "state" : 1 }
  most recently active mongoses:
        "4.4.6" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                34 : Success
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                myshardrs01     990
                                myshardrs02     34
                        too many chunks to print, use verbose if you want to force print

移除分片

shell 复制代码 
use db
db.runCommand({removeShard: "myshardrs02"})

  1. 开启分片功能

    sh.enableSharding("库名")

    sh.shardCollection("库名.集合名",{"key":1})

  2. 插入数据测试

shell 复制代码 
mongos> use test
switched to db test
mongos> for(var i=1;i<=1000;i++){db.comment.insert({_id:i+"",nickname:"Test"+i})}
WriteResult({ "nInserted" : 1 })
mongos> db.comment.count()
1000

可以分别登录到两个分片节点上查看文档数量

  1. 再添加一个路由节点

    步骤同上

  2. 其他可参考的博客

https://www.cnblogs.com/misakivv/p/18171124

https://juejin.cn/post/7160298673182605343
https://mp.weixin.qq.com/s/5oLjCRbXfWNFwmB6jkykZQ

相关推荐
BestandW1shEs13 分钟前
谈谈Mysql的常见基础问题
数据库·mysql
重生之Java开发工程师15 分钟前
MySQL中的CAST类型转换函数
数据库·sql·mysql
教练、我想打篮球17 分钟前
66 mysql 的 表自增长锁
数据库·mysql
Ljw...19 分钟前
表的操作（MySQL）
数据库·mysql·表的操作
哥谭居民000119 分钟前
MySQL的权限管理机制--授权表
数据库
wqq_99225027729 分钟前
ssm旅游推荐系统的设计与开发
数据库·旅游
难以触及的高度1 小时前
mysql中between and怎么用
数据库·mysql
Jacky（易小天）1 小时前
MongoDB比较查询操作符中英对照表及实例详解
数据库·mongodb·typescript·比较操作符
Karoku0662 小时前
【企业级分布式系统】ELK优化
运维·服务器·数据库·elk·elasticsearch
热门推荐
01【音视频 | opus】opus编解码库(opus-1.4)详细介绍以及使用——附带解码示例代码02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04ICLR24大模型提示(1/11) | BadChain：大型语言模型的后门思维链提示05Coze扣子平台完整体验和实践（附国内和国际版对比）06(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明07【postgresql 基础入门】多表联合查询 join与union 并，交，差等集合操作，两者的区别之处08玄机平台应急响应—webshell查杀09Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO10IT专业入门，高考假期预习指南