Oracle Exadata的SCAN监听程序异常重启问题分析

上午遇到一个情况,通过SCAN监听访问数据库时候,长时间无反应,大约2、3分钟后报错丢失连接ORA-12547: TNS:lost contact ;

常规思路下,遇到次问题,有一系列的排查、排除方向,去定位和处理问题。

本次的情况,按照对系统的了解,结合N年的DBA经验,快速定位问题并解决;过程如下:

1、在这个Oracle Exadata环境,有多个数据库,未有人反馈问题,初步判断大部分系统是正常的。看集群状态,运行正常;基本的磁盘空间、TOP命令输出均正常。

2、SCAN监听问题,用了2、3个数据库的SERVICE_NAME去连,都有问题,判断为SCAN监听问题,不是个别数据库问题。

3、用VIP进行连接,连接、查询均正常。

4、查看SCAN 监听状态,lsnrctl status LISTENER_SCAN1执行后发现需要很长时间才可以显示;里面数据库服务名状态都是READY,仔细看Uptime,说明刚重启过,Uptime 0 days 0 hr. 1 min. 5 sec,

5、查看SCAN监听程序日志大小,特别多;直觉反映应该是连接频繁,用监听日志分析的命令 awr系列命令,发现没问题;直接打开监听日志看几千行,问题就清楚了,有大量的同一个IP地址(IP已经脱敏)的连接信息报错--无SID,连接信息是无效的:29-FEB-2024 10:14:57 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.26.32.193)(PORT=42454)) * establish * <unknown sid> *,大量连接导致监听程序异常:

29-FEB-2024 10:13:50 * 12540

TNS-12540: TNS:internal limit restriction exceeded

29-FEB-2024 10:13:50 * 12582

TNS-12582: TNS:invalid operation

TNS-12615: TNS:preempt error

29-FEB-2024 10:13:50 * 1159

TNS-01159: Internal connection limit has been reached; listener has shut down

TNS-12540: TNS:internal limit restriction exceeded

6、把IP发给对应软件公司,确定为一台备份机;停止程序后,监听日志查看正常不再有高频连接,此后SCAN监听未发生过重启,系统恢复正常。

对应的排查国产及日志如下:

1、监听程序异常时候的日志:

复制代码
[grid@hahaha trace]$ tail -f listener_scan1.log 

9-FEB-2024 10:13:26 * service_update * czqcglxt2 * 0
Thu Feb 29 10:13:34 2024
29-FEB-2024 10:13:34 * service_update * less1 * 0
29-FEB-2024 10:13:34 * service_update * hzwsali2 * 0
Thu Feb 29 10:13:50 2024
29-FEB-2024 10:13:50 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=39854)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:13:50 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.26.32.193)(PORT=39856)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:13:50 * 12540
TNS-12540: TNS:internal limit restriction exceeded
29-FEB-2024 10:13:50 * 12582
TNS-12582: TNS:invalid operation
 TNS-12615: TNS:preempt error
29-FEB-2024 10:13:50 * 1159
TNS-01159: Internal connection limit has been reached; listener has shut down
 TNS-12540: TNS:internal limit restriction exceeded
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.1.1)(PORT=1521)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Thu Feb 29 10:13:51 2024
System parameter file is /u01/app/11.2.0.4/grid/network/admin/listener.ora
Log messages written to /u01/app/11.2.0.4/grid/log/diag/tnslsnr/haha/listener_scan1/alert/log.xml
Trace information written to /u01/app/11.2.0.4/grid/log/diag/tnslsnr/haha/listener_scan1/trace/ora_211518_140567299258112.trc
Trace level is currently 0

Started with pid=211518
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Listener completed notification to CRS on start

TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
29-FEB-2024 10:13:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=hahahaha)(USER=grid))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER_SCAN1)(VERSION=186647552)) * status * 0
29-FEB-2024 10:13:51 * version * 0
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.1.1)(PORT=1521)))
29-FEB-2024 10:13:52 * service_register * LsnrAgt * 0
29-FEB-2024 10:13:52 * service_register * lxxxxxxgl1 * 0
............
29-FEB-2024 10:09:49 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=60072)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:09:49 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=60074)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred

2、查看确认一下对应IP的高频率连接问题

情况监听日志,几分钟后的查看

grid@HAHA trace\]$ cat listener_scan1.log \|grep 192.1.1.1\|wc -l 19329 3、根据有问题的IP,找软件维护人员查看,关闭对应程序后,系统恢复。

相关推荐
Java小白程序员21 小时前
MyBatis基础到高级实践:全方位指南(中)
数据库·mybatis
Monly2121 小时前
人大金仓:merge sql error, dbType null, druid-1.2.20
数据库·sql
不宕机的小马达21 小时前
【Mysql|第一篇】Mysql的安装与卸载、Navicat工具的使用
数据库·mysql
float_六七21 小时前
数据库连接池:性能优化的秘密武器
数据库·oracle·性能优化
码界奇点1 天前
MongoDB vs MySQLNoSQL与SQL数据库的架构差异与选型指南
数据库·sql·mongodb·系统架构
IT 小阿姨(数据库)1 天前
PgSQL中pg_stat_user_tables 和 pg_stat_user_objects参数详解
linux·运维·数据库·sql·postgresql·oracle
倔强的石头_1 天前
Windows系统下KingbaseES数据库保姆级安装教程(附常见问题解决)
数据库
麦兜*1 天前
MongoDB 常见错误解决方案:从连接失败到主从同步问题
java·数据库·spring boot·redis·mongodb·容器
RestCloud1 天前
PostgreSQL大表同步优化:如何避免网络和内存瓶颈?
前端·数据库·api
阿里云大数据AI技术1 天前
淘宝闪购基于Flink&Paimon的Lakehouse生产实践:从实时数仓到湖仓一体化的演进之路
数据库·flink