Oracle Exadata的SCAN监听程序异常重启问题分析

上午遇到一个情况,通过SCAN监听访问数据库时候,长时间无反应,大约2、3分钟后报错丢失连接ORA-12547: TNS:lost contact ;

常规思路下,遇到次问题,有一系列的排查、排除方向,去定位和处理问题。

本次的情况,按照对系统的了解,结合N年的DBA经验,快速定位问题并解决;过程如下:

1、在这个Oracle Exadata环境,有多个数据库,未有人反馈问题,初步判断大部分系统是正常的。看集群状态,运行正常;基本的磁盘空间、TOP命令输出均正常。

2、SCAN监听问题,用了2、3个数据库的SERVICE_NAME去连,都有问题,判断为SCAN监听问题,不是个别数据库问题。

3、用VIP进行连接,连接、查询均正常。

4、查看SCAN 监听状态,lsnrctl status LISTENER_SCAN1执行后发现需要很长时间才可以显示;里面数据库服务名状态都是READY,仔细看Uptime,说明刚重启过,Uptime 0 days 0 hr. 1 min. 5 sec,

5、查看SCAN监听程序日志大小,特别多;直觉反映应该是连接频繁,用监听日志分析的命令 awr系列命令,发现没问题;直接打开监听日志看几千行,问题就清楚了,有大量的同一个IP地址(IP已经脱敏)的连接信息报错--无SID,连接信息是无效的:29-FEB-2024 10:14:57 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.26.32.193)(PORT=42454)) * establish * <unknown sid> *,大量连接导致监听程序异常:

29-FEB-2024 10:13:50 * 12540

TNS-12540: TNS:internal limit restriction exceeded

29-FEB-2024 10:13:50 * 12582

TNS-12582: TNS:invalid operation

TNS-12615: TNS:preempt error

29-FEB-2024 10:13:50 * 1159

TNS-01159: Internal connection limit has been reached; listener has shut down

TNS-12540: TNS:internal limit restriction exceeded

6、把IP发给对应软件公司,确定为一台备份机;停止程序后,监听日志查看正常不再有高频连接,此后SCAN监听未发生过重启,系统恢复正常。

对应的排查国产及日志如下:

1、监听程序异常时候的日志:

复制代码
[grid@hahaha trace]$ tail -f listener_scan1.log 

9-FEB-2024 10:13:26 * service_update * czqcglxt2 * 0
Thu Feb 29 10:13:34 2024
29-FEB-2024 10:13:34 * service_update * less1 * 0
29-FEB-2024 10:13:34 * service_update * hzwsali2 * 0
Thu Feb 29 10:13:50 2024
29-FEB-2024 10:13:50 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=39854)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:13:50 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.26.32.193)(PORT=39856)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:13:50 * 12540
TNS-12540: TNS:internal limit restriction exceeded
29-FEB-2024 10:13:50 * 12582
TNS-12582: TNS:invalid operation
 TNS-12615: TNS:preempt error
29-FEB-2024 10:13:50 * 1159
TNS-01159: Internal connection limit has been reached; listener has shut down
 TNS-12540: TNS:internal limit restriction exceeded
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.1.1)(PORT=1521)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Thu Feb 29 10:13:51 2024
System parameter file is /u01/app/11.2.0.4/grid/network/admin/listener.ora
Log messages written to /u01/app/11.2.0.4/grid/log/diag/tnslsnr/haha/listener_scan1/alert/log.xml
Trace information written to /u01/app/11.2.0.4/grid/log/diag/tnslsnr/haha/listener_scan1/trace/ora_211518_140567299258112.trc
Trace level is currently 0

Started with pid=211518
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Listener completed notification to CRS on start

TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
29-FEB-2024 10:13:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=hahahaha)(USER=grid))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER_SCAN1)(VERSION=186647552)) * status * 0
29-FEB-2024 10:13:51 * version * 0
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.1.1)(PORT=1521)))
29-FEB-2024 10:13:52 * service_register * LsnrAgt * 0
29-FEB-2024 10:13:52 * service_register * lxxxxxxgl1 * 0
............
29-FEB-2024 10:09:49 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=60072)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:09:49 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=60074)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred

2、查看确认一下对应IP的高频率连接问题

情况监听日志,几分钟后的查看

grid@HAHA trace\]$ cat listener_scan1.log \|grep 192.1.1.1\|wc -l 19329 3、根据有问题的IP,找软件维护人员查看,关闭对应程序后,系统恢复。

相关推荐
MarkHard1234 小时前
如何利用redis使用一个滑动窗口限流
数据库·redis·缓存
island13145 小时前
【Redis#10】渐进式遍历 | 数据库管理 | redis_cli | RES
数据库·redis·bootstrap
心想事成的幸运大王5 小时前
Redis的过期策略
数据库·redis·缓存
倔强的石头_5 小时前
CentOS 上安装KingbaseES(ISO包)详细教程
数据库
2401_897930066 小时前
使用Docker轻松部署Neo4j图数据库
数据库·docker·neo4j
诗句藏于尽头6 小时前
Django模型与数据库表映射的两种方式
数据库·python·django
寻星探路7 小时前
数据库造神计划第六天---增删改查(CRUD)(2)
java·大数据·数据库
盖世英雄酱581368 小时前
Read timed out问题 排查
java·数据库·后端
云动雨颤9 小时前
Spring Boot配置优化:Tomcat+数据库+缓存+日志,全场景教程
数据库·spring boot·tomcat
RestCloud9 小时前
Kafka实时数据管道:ETL在流式处理中的应用
数据库·kafka·api