oracle 中的大页问题
今天巡检数据库的时候，发现有一个备库节点上的负载很大。这个备库基本无人使用。仅做延时查询使用，按理来说不应该有这么大负载，
环境说明 11.2.0.4 单节点dg库。
问题 大页不生效。且数据库所在服务器负载极大
top - 09:33:00 up 326 days, 19 ``min``, 3 users, ``load average: 17.69, 11.65, 5.83 ``--负载说明
Tasks: 357 total, 1 running, 356 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.3%sy, 0.0%ni, 68.4%id, 31.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65970692k total, 64774768k used, 1195924k ``free``, 6992k buffers
Swap: 33054712k total, 5563048k used, 27491664k ``free``, 399336k cached
``179 root 20 0 0 0 0 D 0.3 0.0 517:34.66 kswapd0
查询当前大页大小：
grep HugePages /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 9960
HugePages_Free: 9960
HugePages_Rsvd: 0
HugePages_Surp: 0
数据库中sga大小
SQL> show parameter sga
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
lock_sga boolean ``FALSE
pre_page_sga boolean ``FALSE
sga_max_size big ``integer 31G
sga_target big ``integer 31G
看到这里问题有点明朗了，好像是数据库没有使用大页导致的。满心欢喜的去查询了一下大页修改的方法。
根据如下 shell脚本跑出了一个修改值。
#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script ``to compute ``values for the
# recommended HugePages/HugeTLB configuration
#
# Note: This script does calculation ``for all shared memory
# segments available ``when the script ``is run, ``no matter it
# ``is an Oracle RDBMS shared memory segment ``or not``.
#
# This script ``is provided ``by Doc ID 401749.1 ``from My Oracle Support
# http://support.oracle.com
# Welcome text
echo ``"
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments. Before proceeding with the execution please note following:
``* For ASM instance, it needs to configure ASMM instead of AMM.
``* The 'pga_aggregate_target' is outside the SGA and
``you should accommodate this while calculating SGA size.
``* In case you changes the DB SGA size,
``as the new SGA will not fit in the previous HugePages configuration,
``it had better disable the whole HugePages,
``start the DB with new SGA size and run the script again.
And make sure that:
``* Oracle Database instance(s) are up and running
``* Oracle Database 11g Automatic Memory Management (AMM) is not setup
``(See Doc ID 749851.1)
``* The shared memory segments can be listed by command:
``# ipcs -m
Press Enter to proceed..."
read
# ``Check for the kernel version
KERN=`uname -r | awk -F. ```'{ printf("%d.%d\n",$1,$2); }'````
# Find ``out the HugePage ``size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk ```'{print $2}'````
if [ -z ``"$HPG_SZ" ];``then
``echo ``"The hugepages may not be supported in the system where the script is being executed."
``exit 1
fi
# Initialize the counter
NUM_PG=0
# Cumulative number ``of pages required ``to handle the running shared memory segments
for SEG_BYTES ``in ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"````
do
```MIN_PG=echo ```"SEG_BYTES/(HPG_SZ*1024)" ``| bc -q ``
``if [ $MIN_PG -gt 0 ]; ``then
```NUM_PG=echo ```"NUM_PG+MIN_PG+1" ``| bc -q ``
``fi
done
RES_BYTES=`echo ```"$NUM_PG * $HPG_SZ * 1024"` | bc -q` ``
# An SGA less than 100MB does ``not make sense
# Bail ``out if that ``is the ``case
if [ $RES_BYTES -lt 100000000 ]; ``then
``echo ``"***********"
``echo ``"** ERROR **"
``echo ``"***********"
``echo ``"Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:
``# ipcs -m
of a size that can match an Oracle Database SGA. Please make sure that:
``* Oracle Database instance is up and running
``* Oracle Database 11g Automatic Memory Management (AMM) is not configured"
``exit 1
fi
# Finish ``with results
case $KERN ``in '2.2'``) echo ``"Kernel version $KERN is not supported. Exiting." ;;
``'2.4'```) HUGETLB_POOL=echo ```"NUM_PG*HPG_SZ/1024" ``| bc -q; ``
``echo ``"Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
``'2.6'``) echo ``"Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
esac
# ``End
然后修改 /etc/sysctl.conf
vm.nr_hugepages = 20000
sysctl -p ``--应用参数 到这里也没有任何问题。
修改完成之后重启数据库
recover managed standby ``database cancel;
shutdown immediate;
startup
启动之后继续查询大页的参数
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 9960
HugePages_Free: 9960
HugePages_Rsvd: 0
HugePages_Surp: 0
发现居然没有改变。。。
因为这个是一个测试库，于是重启了一下 再次查询
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo
AnonHugePages: 176128 kB
HugePages_Total: 20000
HugePages_Free: 20000
HugePages_Rsvd: 0
HugePages_Surp: 0
发现变HugePages_Total是变过来了。但是大页依旧没有使用上。
再次查询数据库相关的参数
alter system reset memory_target scope=spfile;
alter system reset memory_max_target scope=spfile;
SQL> show parameter memory_target
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
memory_target big ``integer 0
SQL> show parameter memory_max_target
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
memory_max_target big ``integer 0
值也都是正常的，最后使出大招修改这个参数
alter system ``set use_large_pages=``only scope=spfile; 让数据库强制使用 大页。重启数据库。
重启时报错了：
[oracle@itunesdg4 ~]$ sqlplus / ``as sysdba
SQL*Plus: Release 11.2.0.4.0 Production ``on Sat Feb 24 14:19:35 2018
Copyright (c) 1982, 2013, Oracle. ``All rights reserved.
Connected ``to an idle instance.
SQL> startup
ORA-27137: unable ``to allocate large pages ``to create a shared memory segment
Linux-x86_64 Error: 1: Operation ``not permitted
Additional information: 201326592
Additional information: 1
上网查询了一下解决方法。
When a process uses ``some memory, the CPU ``is marking the RAM ``as used ``by that process. ``For efficiency, the CPU allocate RAM ``by chunks ``of 4K bytes (it``'s the default value on many platforms). Those chunks are named pages. Those pages can be swapped to disk, etc.
Since the process address space are virtual, the CPU and the operating system have to remember which page belong to which process, and where it is stored. Obviously, the more pages you have, the more time it takes to find where the memory is mapped. When a process uses 1GB of memory, that'``s 262144 entries ``to look up (1GB / 4K). If one Page ``Table Entry consume 8bytes, that's 2MB (262144 * 8) ``to look-up.
Most ``current CPU architectures support bigger pages (so the CPU/OS have less entries ``to look-up), those are named Huge pages (``on Linux), Super Pages (``on BSD) ``or Large Pages (``on Windows), but it ``all the same thing.
1: [root@gsp cdrom]# id oracle
2: uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
3: [root@gsp cdrom]# more /proc/sys/vm/hugetlb_shm_group
4: 0
5: [root@gsp cdrom]# echo 502 >/proc/sys/vm/hugetlb_shm_group
按照网上给出的解决方法去设置了一下这个文件的值。
[oracle@itunesdg4 ~]$ id
uid=502(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
[oracle@itunesdg4 ~]$ echo 502 >/proc/sys/vm/hugetlb_shm_group
-bash: /proc/sys/vm/hugetlb_shm_group: Permission denied
[oracle@itunesdg4 ~]$ su -
Password``:
[root@itunesdg4 ~]# echo 502 >/proc/sys/vm/hugetlb_shm_group
[root@itunesdg4 ~]# exit
logout
[oracle@itunesdg4 ~]$ more /proc/sys/vm/hugetlb_shm_group
502
重启数据库 查询大页使用情况
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo
AnonHugePages: 188416 kB
HugePages_Total: 20000
HugePages_Free: 17965
HugePages_Rsvd: 13838
HugePages_Surp: 0
发现能使用了。
那么究竟是 ``alter system ``set use_large_pages=``only scope=spfile; 这个参数导致了大页生效还是， hugetlb_shm_group 这个文件导致了大页生效呢。做如下测试;
SQL> ``alter system ``set use_large_pages=``true scope=spfile;
SQL> shutdown abort; ``--我这里是测试库 而且是dg库所以直接abort 快速关闭。各位请勿模仿
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo ``--这里数据关闭所以释放了大页
AnonHugePages: 174080 kB
HugePages_Total: 20000
HugePages_Free: 20000
HugePages_Rsvd: 0
HugePages_Surp: 0
SQL> startup ``--重新启动
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo ``--大页还是能正常使用
AnonHugePages: 182272 kB
HugePages_Total: 20000
HugePages_Free: 18566
HugePages_Rsvd: 14439
HugePages_Surp: 0
第一个实验做完了，发现并不是 use_large_pages这个参数导致的大页能够使用。那么尝试修改 hugetlb_shm_group 这个文件 重启数据库
[root@itunesdg4 ~]# echo 0 >/proc/sys/vm/hugetlb_shm_group ``--修改文件的值
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo ``--修改之后查询正常使用大页
AnonHugePages: 186368 kB
HugePages_Total: 20000
HugePages_Free: 17977
HugePages_Rsvd: 13850
HugePages_Surp: 0
SQL> shutdown abort; ``--重启
SQL> startup
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo ``--这里并没有使用大页了。
AnonHugePages: 182272 kB
HugePages_Total: 20000
HugePages_Free: 20000
HugePages_Rsvd: 0
HugePages_Surp: 0
果然是因为这个文件的缘故，那么这个文件究竟是干什么用的呢
网上查询可知：
hugetlb_shm_group
hugetlb_shm_group ``contains group id that ``is allowed ``to create SysV shared memory segment using hugetlb page
该文件表示允许使用hugetlb页创建System VIPC共享内存段的系统组ID。
且这个文件在重启之后会重置成0，所以需要在 /etc/sysctl.conf 中 添加一行 vm.hugetlb_shm_group = 502
最后总结一下大页的配置步骤。
1.查询是否开启了大页
[oracle@itunesdg4 ~]$ cat /proc/meminfo |grep Huge ``--如果total 为0则未开启大页。
AnonHugePages: 196608 kB
HugePages_Total: 20000
HugePages_Free: 17965
HugePages_Rsvd: 13838
HugePages_Surp: 0
2.如果未开启，可以用上面的计算脚本算出一个系统给出的大页值。
3.在/etc/sysctl.conf 中添加一行 vm.nr_hugepages= 计算出的值
4.刷新sysctl.conf 这个文件 sysctl -p 并查询 大页的total是否为你设置的值，如果还是0多刷新几次。
5.设置内存锁：内存锁的数量要大于所设置的大页数量。可以设置为 -1 ，不限制。
vi /etc/security/limits.conf
oracle soft memlock -1
oracle hard memlock -1
6.重启数据库 查询 cat /proc/meminfo |grep Huge 如果 ``free``的值再减少则说明大页正在被使用。如果没使用，或许按照我的方法能给你们提供帮助。