运维项目：企业级四层+七层负载均衡架构搭建

一、项目概述

负责搭建一套支持 10万+并发的高可用负载均衡架构，实现从四层到七层的全栈流量分发。通过 LVS 处理四层 TCP 流量，HAProxy 做中间层 TCP 负载均衡，Nginx 处理七层 HTTP 业务，Keepalived 保障整体高可用。

二、各组件作用说明

|------------|---------|--------------------|--------------------|
| 组件 | 层级 | 核心作用 | 关键特性 |
| LVS | 四层 (L4) | 内核级高性能负载均衡，处理入口流量 | DR模式、10Gbps+吞吐、无单点 |
| Keepalived | 控制层 | VRRP协议实现VIP漂移，健康检查 | 主备切换<<1秒、自动故障转移 |
| HAProxy | 四层 (L4) | TCP连接负载均衡、会话保持 | 连接复用、SSL终止、10万+并发 |
| Nginx | 七层 (L7) | HTTP反向代理、静态资源缓存 | 动静分离、Gzip压缩、缓存策略 |

流量路径：

用户 → VIP(192.168.10.100) → LVS(DR) → HAProxy(TCP) → Nginx(HTTP) → Web应用

三、完整配置代码

1. LVS Director 配置（Master: 192.168.10.10）

安装依赖：

bash 复制代码

# CentOS 7
yum install -y ipvsadm keepalived

# 加载内核模块
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack

# 开机自动加载
echo "ip_vs" >> /etc/modules-load.d/lvs.conf
echo "ip_vs_rr" >> /etc/modules-load.d/lvs.conf

配置虚拟IP（VIP）：

bash 复制代码

# /etc/sysconfig/network-scripts/ifcfg-eth0:0
DEVICE=eth0:0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.10.100
NETMASK=255.255.255.255

LVS 规则配置：

bash 复制代码

#!/bin/bash
# /usr/local/bin/lvs_dr.sh

VIP=192.168.10.100
RIP1=192.168.10.20
RIP2=192.168.10.21
RIP3=192.168.10.22

# 清除旧规则
ipvsadm -C

# 添加虚拟服务，使用WRR算法
ipvsadm -A -t $VIP:80 -s wrr

# 添加真实服务器（HAProxy节点），DR模式
ipvsadm -a -t $VIP:80 -r $RIP1:80 -g -w 1
ipvsadm -a -t $VIP:80 -r $RIP2:80 -g -w 1
ipvsadm -a -t $VIP:80 -r $RIP3:80 -g -w 1

# 保存规则
ipvsadm -S > /etc/sysconfig/ipvsadm

echo "LVS-DR 配置完成"

Keepalived 主节点配置：

bash 复制代码

# /etc/keepalived/keepalived.conf

global_defs {
    router_id LVS_MASTER
    notification_email {
        ops@company.com
    }
    notification_email_from lvs@company.com
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
}

# VRRP 实例 - 实现VIP高可用
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100          # 主节点优先级高
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1234
    }
    virtual_ipaddress {
        192.168.10.100/24
    }
    # 状态切换通知脚本
    notify_master "/usr/local/bin/notify.sh master"
    notify_backup "/usr/local/bin/notify.sh backup"
    notify_fault "/usr/local/bin/notify.sh fault"
}

# LVS 虚拟服务器配置
virtual_server 192.168.10.100 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR              # 直接路由模式
    protocol TCP
    
    # 后端真实服务器（HAProxy节点）
    real_server 192.168.10.20 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.10.21 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
    real_server 192.168.10.22 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }
}

状态切换通知脚本：

bash 复制代码

#!/bin/bash
# /usr/local/bin/notify.sh

TYPE=$1
HOST=$(hostname)
TIME=$(date '+%Y-%m-%d %H:%M:%S')

# 记录日志
echo "[$TIME] $HOST changed to $TYPE" >> /var/log/keepalived.log

# 发送告警（可选）
if [ "$TYPE" == "master" ]; then
    # 可以接入企业微信/钉钉告警
    echo "VIP已漂移到 $HOST" | mail -s "Keepalived Master切换" ops@company.com
fi

2. LVS Director 配置（Backup: 192.168.10.11）

bash 复制代码

# /etc/keepalived/keepalived.conf

global_defs {
    router_id LVS_BACKUP
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90              # 备节点优先级低
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1234
    }
    virtual_ipaddress {
        192.168.10.100/24
    }
}

# LVS 虚拟服务器配置（与主节点相同）
virtual_server 192.168.10.100 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    protocol TCP
    
    real_server 192.168.10.20 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 192.168.10.21 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 192.168.10.22 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

3. HAProxy Real Server 配置（192.168.10.20/21/22）

抑制ARP响应（DR模式必需）：

bash 复制代码

#!/bin/bash
# /usr/local/bin/lvs_rs.sh

VIP=192.168.10.100

# 绑定VIP到lo接口（仅用于响应，不对外宣告）
ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP up

# 抑制ARP响应，避免VIP冲突
echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce

# 添加路由
route add -host $VIP dev lo:0

echo "Real Server LVS配置完成"

安装 HAProxy：

bash 复制代码

yum install -y haproxy

# 备份默认配置
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak

HAProxy 配置文件：

bash 复制代码

# /etc/haproxy/haproxy.cfg

#==============================================================================
# 全局配置
#==============================================================================
global
    log         127.0.0.1 local2 info
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     50000              # 最大并发连接数
    user        haproxy
    group       haproxy
    daemon
    
    # 性能优化
    nbproc 2                       # 多进程模式
    cpu-map auto:1/1-2 0-1
    
    # SSL性能
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

#==============================================================================
# 默认配置
#==============================================================================
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  forwardfor except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 30000

#==============================================================================
# 四层TCP前端（接收LVS转发的流量）
#==============================================================================
frontend tcp_frontend
    bind *:80
    mode tcp
    default_backend nginx_tcp_backend

#==============================================================================
# 四层TCP后端（负载均衡到Nginx）
#==============================================================================
backend nginx_tcp_backend
    mode tcp
    balance roundrobin        # 轮询算法
    
    # 健康检查
    option tcp-check
    tcp-check connect port 80
    tcp-check send PING\\r\\n
    tcp-check expect string +PONG
    
    # Nginx后端服务器
    server nginx1 192.168.10.30:80 check weight 1 inter 2000 rise 2 fall 3
    server nginx2 192.168.10.31:80 check weight 1 inter 2000 rise 2 fall 3
    server nginx3 192.168.10.32:80 check weight 1 inter 2000 rise 2 fall 3
    server nginx4 192.168.10.33:80 check weight 1 inter 2000 rise 2 fall 3

#==============================================================================
# 统计页面（监控用）
#==============================================================================
listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 30s
    stats admin if TRUE
    stats auth admin:admin123

HAProxy 启动脚本：

bash 复制代码

#!/bin/bash
# /usr/local/bin/haproxy_start.sh

# 检查配置
haproxy -c -f /etc/haproxy/haproxy.cfg
if [ $? -ne 0 ]; then
    echo "HAProxy配置检查失败"
    exit 1
fi

# 启动服务
systemctl start haproxy
systemctl enable haproxy

# 检查状态
systemctl status haproxy
echo "HAProxy启动完成"

4. Nginx 配置（192.168.10.30/31/32/33）

安装 Nginx：

bash 复制代码

# 添加官方源
cat > /etc/yum.repos.d/nginx.repo << 'EOF'
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
EOF

yum install -y nginx

Nginx 主配置：

bash 复制代码

# /etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;           # 自动根据CPU核心数
worker_rlimit_nofile 65535;       # 文件描述符限制

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    use epoll;                    # 高性能事件模型
    worker_connections  65535;    # 单个worker最大连接
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    
    # 日志格式
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      'upstream=$upstream_addr response_time=$request_time';
    
    access_log  /var/log/nginx/access.log  main;
    
    # 性能优化
    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;
    keepalive_timeout  65;
    types_hash_max_size 2048;
    
    # Gzip压缩
    gzip  on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml application/json 
               application/javascript application/xml+rss 
               application/atom+xml image/svg+xml;
    
    # 连接限制
    limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
    limit_req_zone $binary_remote_addr zone=req_limit:10m rate=10r/s;
    
    # 包含虚拟主机配置
    include /etc/nginx/conf.d/*.conf;
}

Nginx 虚拟主机配置：

bash 复制代码

# /etc/nginx/conf.d/default.conf

upstream web_backend {
    least_conn;                   # 最少连接算法
    
    server 192.168.10.40:8080 weight=5 max_fails=3 fail_timeout=30s;
    server 192.168.10.41:8080 weight=5 max_fails=3 fail_timeout=30s;
    
    keepalive 32;                 # 长连接数
}

server {
    listen       80;
    server_name  www.company.com;
    root         /usr/share/nginx/html;
    
    # 限制连接数
    limit_conn conn_limit 20;
    limit_req zone=req_limit burst=20 nodelay;
    
    # 静态资源缓存
    location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff|woff2)$ {
        expires 30d;
        add_header Cache-Control "public, immutable";
        access_log off;
    }
    
    # 反向代理到应用服务器
    location / {
        proxy_pass http://web_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # 传递真实IP
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $host;
        
        # 超时设置
        proxy_connect_timeout 5s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        
        # 缓冲区
        proxy_buffering on;
        proxy_buffer_size 4k;
        proxy_buffers 8 4k;
    }
    
    # 健康检查页面
    location /nginx_status {
        stub_status on;
        access_log off;
        allow 192.168.10.0/24;
        deny all;
    }
    
    # 错误页面
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }
}

5. 监控脚本

LVS 状态监控脚本：

bash 复制代码

#!/bin/bash
# /usr/local/bin/lvs_monitor.sh

VIP="192.168.10.100"
LOG="/var/log/lvs_monitor.log"

echo "========== $(date) ==========" >> $LOG

# 检查VIP是否存在
ip addr show | grep $VIP > /dev/null
if [ $? -eq 0 ]; then
    echo "[OK] VIP $VIP 存在" >> $LOG
else
    echo "[ERROR] VIP $VIP 不存在" >> $LOG
fi

# 检查LVS规则
echo "当前LVS规则：" >> $LOG
ipvsadm -Ln >> $LOG

# 检查后端服务器状态
echo "后端服务器连接数：" >> $LOG
ipvsadm -Ln --stats >> $LOG

# 检查Keepalived状态
systemctl status keepalived >> $LOG 2>&1

HAProxy 健康检查脚本：

bash 复制代码

#!/bin/bash
# /usr/local/bin/haproxy_check.sh

URL="http://localhost:8404/stats"
AUTH="admin:admin123"
LOG="/var/log/haproxy_check.log"

# 获取HAProxy状态
curl -s -u $AUTH $URL | grep -E "nginx|BACKEND" >> $LOG

# 检查后端服务器状态
for server in nginx1 nginx2 nginx3 nginx4; do
    STATUS=$(echo "show servers state" | socat stdio /var/run/haproxy.sock | grep $server | awk '{print $18}')
    if [ "$STATUS" == "2" ]; then
        echo "[OK] $server 状态正常" >> $LOG
    else
        echo "[WARN] $server 状态异常: $STATUS" >> $LOG
    fi
done

Nginx 日志分析脚本：

bash 复制代码

#!/bin/bash
# /usr/local/bin/nginx_log_analyze.sh

LOG="/var/log/nginx/access.log"
REPORT="/tmp/nginx_report.txt"

echo "========== Nginx 访问报告 $(date) ==========" > $REPORT

# 总请求数
echo "总请求数: $(wc -l < $LOG)" >> $REPORT

# 状态码分布
echo -e "\n状态码分布:" >> $REPORT
awk '{print $9}' $LOG | sort | uniq -c | sort -rn >> $REPORT

# 最慢请求TOP10
echo -e "\n最慢请求TOP10:" >> $REPORT
awk '{print $NF, $7}' $LOG | sort -rn | head -10 >> $REPORT

# 404错误TOP10
echo -e "\n404错误TOP10:" >> $REPORT
awk '$9 == 404 {print $7}' $LOG | sort | uniq -c | sort -rn | head -10 >> $REPORT

cat $REPORT

6. 一键部署脚本

bash 复制代码

#!/bin/bash
# /usr/local/bin/deploy_all.sh
# 一键部署负载均衡架构

set -e

echo "========== 开始部署负载均衡架构 =========="

# 1. 配置所有节点的hosts
cat >> /etc/hosts << 'EOF'
192.168.10.10  lvs-master
192.168.10.11  lvs-backup
192.168.10.20  haproxy1
192.168.10.21  haproxy2
192.168.10.22  haproxy3
192.168.10.30  nginx1
192.168.10.31  nginx2
192.168.10.32  nginx3
192.168.10.33  nginx4
192.168.10.40  app1
192.168.10.41  app2
EOF

# 2. 关闭防火墙和SELinux（测试环境）
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

# 3. 时间同步
yum install -y ntpdate
ntpdate -u ntp.aliyun.com
hwclock --systohc

# 4. 内核参数优化
cat >> /etc/sysctl.conf << 'EOF'
# 网络优化
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_tw_buckets = 6000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 65536
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 1024 65535
EOF
sysctl -p

echo "基础环境配置完成"

# 根据角色执行不同配置
case $(hostname) in
    lvs-master|lvs-backup)
        echo "配置LVS Director..."
        bash /usr/local/bin/lvs_dr.sh
        systemctl start keepalived
        systemctl enable keepalived
        ;;
    haproxy*)
        echo "配置HAProxy Real Server..."
        bash /usr/local/bin/lvs_rs.sh
        systemctl start haproxy
        systemctl enable haproxy
        ;;
    nginx*)
        echo "配置Nginx..."
        systemctl start nginx
        systemctl enable nginx
        ;;
    *)
        echo "未知节点类型"
        ;;
esac

echo "========== 部署完成 =========="

四、简历项目描述

企业级高可用负载均衡架构搭建

独立搭建 LVS(DR) + Keepalived + HAProxy + Nginx 四层+七层负载均衡架构，支持 10万+ 并发连接

使用 LVS-DR 模式实现内核级四层负载均衡，通过 Keepalived VRRP 协议实现 VIP 高可用，故障切换时间 < 1秒

配置 HAProxy 进行 TCP 层负载均衡，实现连接复用和健康检查，后端 Nginx 处理七层 HTTP 业务

编写自动化部署脚本，实现一键环境配置和服务部署，减少人工操作时间 80%

设计日志分析和监控告警脚本，实现服务状态实时监控和故障快速定位

技术栈：CentOS 7、LVS、Keepalived、HAProxy、Nginx、Shell、TCP/IP