一、现象
操作系统内存使用率告警，已达到98%,，告警内容如下：
【全景监控：Oracle主机内存使用监控】
【主机名】：XXXXX11
【主机IP】主机IP：*.126.15
【告警内容】当前内存使用率为98.9%，超警戒值90%
请及时处理
二、分析
1、查看操作系统内存使用情况
free -g
top
M
2、查看sga和pga分配的内存分别是：140g和40g。
3、集群日志发现如下报错
2022-03-20 20:22:19.247:
[ohasd(30064)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /u01/app/11.2.0/grid/auth/ohasd/gw11/A1076079
2022-03-20 20:22:19.248:
[ohasd(30064)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /u01/app/11.2.0/grid/auth/ohasd/gw11/A0906893
4、查看内存排行榜，top M ohas.d占用率最高
5、根据alert的报错日志查看空间使用率77%
[grid@gw11 crsd]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_user01-LogVol04
50G 4.5G 43G 10% /
tmpfs 126G 736M 126G 1% /dev/shm
/dev/sda1 190M 42M 139M 24% /boot
/dev/mapper/vg_user01-LogVol01
20G 45M 19G 1% /tmp
/dev/mapper/vg_user01-LogVol02
148G 108G 33G 77% /u01
/dev/mapper/vg_user01-LogVol00
9.8G 636M 8.6G 7% /var
/dev/mapper/vg_user01-ogg_vg
493G 8.8G 459G 2% /ogg
/dev/sds 2.0T 76G 1.8T 4% /nfs
6、当前空间看并没有不足，进一步查看删除未释放的空间，发现异常，存在大量删除未释放的文件，至此问题原因已明确，由于在删除文件的时候，进程占用，导致空间未能释放，从而引起空间不足。
[root@gw11 oraagent_grid]# lsof |grep delete
oracle 47876 oracle 3u REG 253,5 7607 21495944 /ogg/dirrpt/R_P_JF9.rpt (deleted)
oracle 47876 oracle 5w REG 253,5 10490641 114 /ogg/ggserr.log.9 (deleted)
oracle 47876 oracle 8w REG 253,5 5876 21495906 /ogg/dirrpt/r_p_jf9.dsc (deleted)
server 246950 oracle 2w REG 253,5 10490505 119 /ogg/ggserr.log.9 (deleted)
server 246951 oracle 2w REG 253,5 10490505 119 /ogg/ggserr.log.9 (deleted)
server 246952 oracle 2w REG 253,5 10490505 119 /ogg/ggserr.log.9 (deleted)
server 246953 oracle 2w REG 253,5 10490505 119 /ogg/ggserr.log.9 (deleted)
oracle 257774 oracle 3u REG 253,5 7310 21496010 /ogg/dirrpt/REP_FLO39.rpt (deleted)
oracle 257774 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257774 oracle 8w REG 253,5 4855 21495982 /ogg/dirrpt/rep_flo39.dsc (deleted)
oracle 257775 oracle 3u REG 253,5 7308 21496000 /ogg/dirrpt/REP_FLO29.rpt (deleted)
oracle 257775 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257775 oracle 8w REG 253,5 4856 21495932 /ogg/dirrpt/rep_flo29.dsc (deleted)
oracle 257776 oracle 3u REG 253,5 7310 21496013 /ogg/dirrpt/REP_FLO59.rpt (deleted)
oracle 257776 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257776 oracle 8w REG 253,5 5003 21495861 /ogg/dirrpt/rep_flo59.dsc (deleted)
oracle 257777 oracle 3u REG 253,5 7491 21496015 /ogg/dirrpt/REP_ORD19.rpt (deleted)
oracle 257777 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257777 oracle 8w REG 253,5 5662 21496050 /ogg/dirrpt/REP_ORD19.dsc (deleted)
oracle 257778 oracle 3u REG 253,5 6482 21495827 /ogg/dirrpt/REP_FLO19.rpt (deleted)
oracle 257778 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257778 oracle 8w REG 253,5 5169 21495994 /ogg/dirrpt/rep_flo19.dsc (deleted)
oracle 257779 oracle 3u REG 253,5 7474 21496020 /ogg/dirrpt/REP_ORD29.rpt (deleted)
oracle 257779 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257779 oracle 8w REG 253,5 5522 21495877 /ogg/dirrpt/REP_ORD29.dsc (deleted)
oracle 257780 oracle 3u REG 253,5 7474 21496028 /ogg/dirrpt/REP_ORD59.rpt (deleted)
oracle 257780 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257780 oracle 8w REG 253,5 5498 21496153 /ogg/dirrpt/REP_ORD59.dsc (deleted)
oracle 257781 oracle 3u REG 253,5 6481 21496014 /ogg/dirrpt/REP_FLOW9.rpt (deleted)
oracle 257781 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257781 oracle 8w REG 253,5 4840 21495993 /ogg/dirrpt/rep_flow9.dsc (deleted)
oracle 257782 oracle 3u REG 253,5 7310 21496012 /ogg/dirrpt/REP_FLO49.rpt (deleted)
oracle 257782 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257782 oracle 8w REG 253,5 5099 21495977 /ogg/dirrpt/rep_flo49.dsc (deleted)
oracle 257783 oracle 3u REG 253,5 7475 21496046 /ogg/dirrpt/REP_ORDE9.rpt (deleted)
oracle 257783 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257783 oracle 8w REG 253,5 5662 21496148 /ogg/dirrpt/rep_orde9.dsc (deleted)
oracle 257785 oracle 3u REG 253,5 7474 21496023 /ogg/dirrpt/REP_ORD39.rpt (deleted)
oracle 257785 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257785 oracle 8w REG 253,5 5661 21496212 /ogg/dirrpt/REP_ORD39.dsc (deleted)
oracle 257786 oracle 3u REG 253,5 7474 21496025 /ogg/dirrpt/REP_ORD49.rpt (deleted)
oracle 257786 oracle 5w REG 253,5 10490644 120 /ogg/ggserr.log.9 (deleted)
oracle 257786 oracle 8w REG 253,5 5498 21496152 /ogg/dirrpt/REP_ORD49.dsc (deleted)
oracle 268915 oracle 3u REG 253,5 7755 21496082 /ogg/dirrpt/REP_ICR49.rpt (deleted)
oracle 268915 oracle 5w REG 253,5 10490643 123 /ogg/ggserr.log.9 (deleted)
oracle 268915 oracle 8w REG 253,5 330 21495988 /ogg/dirrpt/REP_ICR49.dsc (deleted)
oracle 268916 oracle 3u REG 253,5 7801 21496080 /ogg/dirrpt/REP_ICR29.rpt (deleted)
oracle 268916 oracle 5w REG 253,5 10490643 123 /ogg/ggserr.log.9 (deleted)
oracle 268916 oracle 8w REG 253,5 330 21495986 /ogg/dirrpt/REP_ICR29.dsc (deleted)
oracle 268917 oracle 3u REG 253,5 7755 21496079 /ogg/dirrpt/REP_ICR19.rpt (deleted)
oracle 268917 oracle 5w REG 253,5 10490643 123 /ogg/ggserr.log.9 (deleted)
oracle 268917 oracle 8w REG 253,5 330 21495987 /ogg/dirrpt/REP_ICR19.dsc (deleted)
oracle 268918 oracle 3u REG 253,5 7755 21496083 /ogg/dirrpt/REP_ICR39.rpt (deleted)
oracle 268918 oracle 5w REG 253,5 10490643 123 /ogg/ggserr.log.9 (deleted)
oracle 268918 oracle 8w REG 253,5 330 21495985 /ogg/dirrpt/REP_ICR39.dsc (deleted)
gpnpd.bin 370856 grid 36uw REG 253,3 0 524808 /var/tmp/.oracle/ora_gipc_GPNPD_bywsgw11_lock (deleted)
gipcd.bin 370870 grid 34uw REG 253,3 0 524816 /var/tmp/.oracle/ora_gipc_gipcd_bywsgw11_lock (deleted)
osysmond. 370883 root 58uw REG 253,3 0 524818 /var/tmp/.oracle/ora_gipc_sbywsgw11gridbyicdb-clusterCRFM_CLIIPC_lock (deleted)
osysmond. 370883 root 60uw REG 253,3 0 524820 /var/tmp/.oracle/ora_gipc_sbywsgw11gridbyicdb-clusterCRFM_SIPC_lock (deleted)
ocssd.bin 370964 grid 35uw REG 253,3 0 524814 /var/tmp/.oracle/sOCSSD_LL_bywsgw11__lock (deleted)
ocssd.bin 370964 grid 94uw REG 253,3 0 524844 /var/tmp/.oracle/sOracle_CSS_LclLstnr_byicdb-cluster_1_lock (deleted)
ocssd.bin 370964 grid 100uw REG 253,3 0 524845 /var/tmp/.oracle/sOCSSD_LL_bywsgw11_byicdb-cluster_lock (deleted)
oracle 372244 grid 16w REG 253,4 832 5657402 /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ping_372244.trc (deleted)
oracle 372244 grid 17w REG 253,4 62 5657404 /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ping_372244.trm (deleted)
oracle 372246 grid 15w REG 253,4 106031 5657393 /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_372246_base_1.trc (deleted)
oracle 372246 grid 16w REG 253,4 13846 5657394 /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_372246_base_1.trm (deleted)
crsd.bin 373237 root 61uw REG 253,3 0 524846 /var/tmp/.oracle/sprocr_local_conn_0_PROC_lock (deleted)
crsd.bin 373237 root 122uw REG 253,3 0 524847 /var/tmp/.oracle/sCRSD_IPC_SOCKET_11_lock (deleted)
oracle 373253 grid 11u REG 253,4 756 5657420 /u01/app/11.2.0/grid/rdbms/audit/+ASM1_ora_373253_20200815180326047345143795.aud (deleted)
oracle 373257 grid 13u REG 253,4 756 5657421 /u01/app/11.2.0/grid/rdbms/audit/+ASM1_ora_373257_20200815180326173317143795.aud (deleted)
oracle 374115 oracle 16w REG 253,4 886 5511353 /u01/app/oracle/diag/rdbms/byicdb/byicdb1/trace/byicdb1_ping_374115.trc (deleted)
oracle 374115 oracle 17w REG 253,4 64 5511356 /u01/app/oracle/diag/rdbms/byicdb/byicdb1/trace/byicdb1_ping_374115.trm (deleted)
oracle 374185 oracle 18w REG 253,4 877 5511337 /u01/app/oracle/diag/rdbms/byicdb/byicdb1/trace/byicdb1_asmb_374185.trc (deleted)
oracle 374185 oracle 19w REG 253,4 64 5511338 /u01/app/oracle/diag/rdbms/byicdb/byicdb1/trace/byicdb1_asmb_374185.trm (deleted)
oracle 374197 grid 13u REG 253,4 758 5657431 /u01/app/11.2.0/grid/rdbms/audit/+ASM1_ora_374197_20200815180410057079143795.aud (deleted)
oracle 399268 oracle 3u REG 253,5 7241 21495844 /ogg/dirrpt/REP_REQE9.rpt (deleted)
oracle 399268 oracle 5w REG 253,5 10490459 124 /ogg/ggserr.log.9 (deleted)
oracle 399268 oracle 8w REG 253,5 740 21495867 /ogg/dirrpt/rep_reqe9.dsc (deleted)
三、解决
1、清理一些必要的无用日志释放空间。
2、重启集群，释放被占用的内存。