用 root 用户进入容器安装 gdb
直接执行(注意 -u 0 就是 root):
docker compose exec -u 0 app sh -lc 'apt-get update && apt-get install -y gdb'
装完确认:
docker compose exec -u 0 app sh -lc 'gdb --version'
docker compose exec -u 0 app sh -lc '
apt-get update &&
apt-get install -y gdb &&
pip install -U memray
'
docker compose exec -u 0 app sh -lc '
apt-get update &&
apt-get install -y gdb &&
/label-studio/.venv/bin/pip install -U memray
'
在容器里面:
sh -lc '
for p in /proc/[0-9]*; do
cmd=$(tr "\0" " " < "$p/cmdline" 2>/dev/null || true)
echo "$cmd" | grep -q uwsgi && echo "PID=$(basename $p) CMD=$cmd"
done | head -20
'
输入上面查询出来的pid:
sh -lc 'cat /proc/<PID>/status | egrep "VmRSS|VmSize|RssAnon|RssFile|RssShmem"'
在宿主机:
docker compose exec app sh -lc '
for p in /proc/[0-9]*; do
cmd=$(tr "\0" " " < "$p/cmdline" 2>/dev/null || true)
echo "$cmd" | grep -q uwsgi && echo "PID=$(basename $p) CMD=$cmd"
done | head -20
'
这会输出类似:
- PID=1 CMD=... uwsgi ...(可能是 master)
- 以及其他 worker 的 PID
2) 看哪个 PID 内存大(在容器内)
把上一步输出的某个 PID(例如 1 或 23)替换进去:
docker compose exec app sh -lc 'cat /proc/<PID>/status | egrep "VmRSS|VmSize|RssAnon|RssFile|RssShmem"'
先解释一下你看到的 PID
- PID=1:基本可以认为是 uWSGI master(容器主进程)
- PID=163:是 uWSGI worker / http进程(真正处理请求的进程之一)
-
PID=722/731:是你临时执行的 sh -lc ... 子进程,命令结束就没了,所以 /proc/... 会消失,这是正常的。
PID=298155 CMD=/label-studio/.venv/bin/python /label-studio/.venv/bin/uwsgi --ini /label-studio/deploy/uwsgi.ini
PID=298632 CMD=/label-studio/.venv/bin/python /label-studio/.venv/bin/uwsgi --ini /label-studio/deploy/uwsgi.ini
[root@localhost bztongyong]# docker compose exec app sh -lc "ps aux | grep uwsgi | head"
sh: 1: ps: not found
[root@localhost bztongyong]# docker compose exec app sh -lc "298155"
sh: 1: 298155: not found
[root@localhost bztongyong]# docker compose exec app sh -lc 298155
sh: 1: 298155: not found
[root@localhost bztongyong]# docker compose exec app sh -lc 'for p in /proc/[0-9]; do
cmd=(tr "\0" " " < "p/cmdline" 2>/dev/null || true)
echo "cmd" | grep -q uwsgi && echo "PID=(basename p) CMD=cmd"
done | head -20
'
PID=1 CMD=/label-studio/.venv/bin/python /label-studio/.venv/bin/uwsgi --ini /label-studio/deploy/uwsgi.ini
PID=163 CMD=/label-studio/.venv/bin/python /label-studio/.venv/bin/uwsgi --ini /label-studio/deploy/uwsgi.ini
PID=722 CMD=sh -lc
for p in /proc/[0-9]; do
cmd=(tr "" " " < "p/cmdline" 2>/dev/null || true)
echo "cmd" | grep -q uwsgi && echo "PID=(basename p) CMD=cmd"
done | head -20PID=731 CMD=sh -lc
for p in /proc/[0-9]*; do
cmd=(tr "" " " < "p/cmdline" 2>/dev/null || true)
echo "cmd" | grep -q uwsgi && echo "PID=(basename p) CMD=cmd"
done | head -20[root@localhost bztongyong]# docker compose exec app sh -lc 'cat /proc/1/status | egrep "VmRSS|VmSize|RssAnon|RssFile|RssShmem"'
VmSize: 23060 kB
VmRSS: 14720 kB
RssAnon: 8788 kB
RssFile: 4824 kB
RssShmem: 1108 kB
[root@localhost bztongyong]# docker compose exec app sh -lc 'cat /proc/163/status | egrep "VmRSS|VmSize|RssAnon|RssFile|RssShmem"'
VmSize: 31256 kB
VmRSS: 17216 kB
RssAnon: 16984 kB
RssFile: 216 kB
RssShmem: 16 kB
[root@localhost bztongyong]# docker compose exec app sh -lc 'cat /proc/722/status | egrep "VmRSS|VmSize|RssAnon|RssFile|RssShmem"'
cat: /proc/722/status: No such file or directory
[root@localhost bztongyong]# docker compose exec app sh -lc 'cat /proc/731/status | egrep "VmRSS|VmSize|RssAnon|RssFile|RssShmem"'
cat: /proc/731/status: No such file or directory
[root@localhost bztongyong]# 看看吧
用 Memray 的正确抓法(抓 worker,不抓 PID=1)
请对 PID=163 做 attach(建议先录 2--5 分钟):
- 第 1 行:确保容器里装了 memray
- pip show memray:检查是否已安装
- || pip install -U memray:如果没装就安装/升级
- 不产生报告文件
- 第 2 行:附加到进程 PID=163,录制 300 秒的内存分配
- memray attach 163:连接到容器内 PID=163(通常是 uWSGI worker)
- -o /tmp/memray-163.bin:把原始采样数据保存到容器内 /tmp/ 目录
- --duration 300:录 300 秒(5 分钟)
- 产物:容器内 /tmp/memray-163.bin
- 第 3 行:把 bin 转成可视化 HTML 报告
- memray flamegraph ... -o ...:生成 flamegraph
- 产物:容器内 /tmp/memray-163.html
- 第 4 行:把容器里的 HTML 拷到宿主机当前目录
- docker compose ps -q app:拿到 app 容器 ID
- docker cp <容器ID>:/tmp/memray-163.html ./memray-163.html
- 产物:宿主机当前目录下 memray-163.html(用浏览器打开看)
先一次性准备环境(只做一次):
docker compose exec -u 0 app sh -lc 'apt-get update && apt-get install -y gdb'
docker compose exec -u 0 app sh -lc '/label-studio/.venv/bin/pip install -U memray'
先执行输出进程号:
docker compose exec app sh -lc '
for p in /proc/[0-9]*; do
cmd=$(tr "\0" " " < "$p/cmdline" 2>/dev/null || true)
echo "$cmd" | grep -q uwsgi && echo "PID=$(basename $p) CMD=$cmd"
done | head -20'
docker compose exec -u 0 app sh -lc '
apt-get update &&
apt-get install -y gdb &&
/label-studio/.venv/bin/pip install -U memray
'
再根据进程号执行下面命令: 选 不是 1 的那个(一般 1 是 master),优先选 worker 的 PID 去 attach。
docker compose exec app sh -lc '/label-studio/.venv/bin/python -m memray attach <PID> -o /tmp/memray.bin --duration 300'
docker compose exec app sh -lc '/label-studio/.venv/bin/python -m memray flamegraph /tmp/memray.bin -o /tmp/memray.html'
docker cp $(docker compose ps -q app):/tmp/memray.html ./memray.html
如果你想将docker compose exec app sh -lc 'memray attach 163 -o /tmp/memray-163.bin --duration 300'
生成的memray-163.bin文件拷到挂载的数据目录(方便下载)执行下面代码
你们有挂载 ./mydata:/un-label/data,可以把报告放进去:
docker compose exec app sh -lc "cp /tmp/memray-163.html /un-label/data/memray-163.html && ls -lh /un-label/data | grep memray"