Oracle Exadata的SCAN监听程序异常重启问题分析

上午遇到一个情况,通过SCAN监听访问数据库时候,长时间无反应,大约2、3分钟后报错丢失连接ORA-12547: TNS:lost contact ;

常规思路下,遇到次问题,有一系列的排查、排除方向,去定位和处理问题。

本次的情况,按照对系统的了解,结合N年的DBA经验,快速定位问题并解决;过程如下:

1、在这个Oracle Exadata环境,有多个数据库,未有人反馈问题,初步判断大部分系统是正常的。看集群状态,运行正常;基本的磁盘空间、TOP命令输出均正常。

2、SCAN监听问题,用了2、3个数据库的SERVICE_NAME去连,都有问题,判断为SCAN监听问题,不是个别数据库问题。

3、用VIP进行连接,连接、查询均正常。

4、查看SCAN 监听状态,lsnrctl status LISTENER_SCAN1执行后发现需要很长时间才可以显示;里面数据库服务名状态都是READY,仔细看Uptime,说明刚重启过,Uptime 0 days 0 hr. 1 min. 5 sec,

5、查看SCAN监听程序日志大小,特别多;直觉反映应该是连接频繁,用监听日志分析的命令 awr系列命令,发现没问题;直接打开监听日志看几千行,问题就清楚了,有大量的同一个IP地址(IP已经脱敏)的连接信息报错--无SID,连接信息是无效的:29-FEB-2024 10:14:57 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.26.32.193)(PORT=42454)) * establish * <unknown sid> *,大量连接导致监听程序异常:

29-FEB-2024 10:13:50 * 12540

TNS-12540: TNS:internal limit restriction exceeded

29-FEB-2024 10:13:50 * 12582

TNS-12582: TNS:invalid operation

TNS-12615: TNS:preempt error

29-FEB-2024 10:13:50 * 1159

TNS-01159: Internal connection limit has been reached; listener has shut down

TNS-12540: TNS:internal limit restriction exceeded

6、把IP发给对应软件公司,确定为一台备份机;停止程序后,监听日志查看正常不再有高频连接,此后SCAN监听未发生过重启,系统恢复正常。

对应的排查国产及日志如下:

1、监听程序异常时候的日志:

复制代码
[grid@hahaha trace]$ tail -f listener_scan1.log 

9-FEB-2024 10:13:26 * service_update * czqcglxt2 * 0
Thu Feb 29 10:13:34 2024
29-FEB-2024 10:13:34 * service_update * less1 * 0
29-FEB-2024 10:13:34 * service_update * hzwsali2 * 0
Thu Feb 29 10:13:50 2024
29-FEB-2024 10:13:50 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=39854)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:13:50 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.26.32.193)(PORT=39856)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:13:50 * 12540
TNS-12540: TNS:internal limit restriction exceeded
29-FEB-2024 10:13:50 * 12582
TNS-12582: TNS:invalid operation
 TNS-12615: TNS:preempt error
29-FEB-2024 10:13:50 * 1159
TNS-01159: Internal connection limit has been reached; listener has shut down
 TNS-12540: TNS:internal limit restriction exceeded
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.1.1)(PORT=1521)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Thu Feb 29 10:13:51 2024
System parameter file is /u01/app/11.2.0.4/grid/network/admin/listener.ora
Log messages written to /u01/app/11.2.0.4/grid/log/diag/tnslsnr/haha/listener_scan1/alert/log.xml
Trace information written to /u01/app/11.2.0.4/grid/log/diag/tnslsnr/haha/listener_scan1/trace/ora_211518_140567299258112.trc
Trace level is currently 0

Started with pid=211518
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
Listener completed notification to CRS on start

TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
29-FEB-2024 10:13:51 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=hahahaha)(USER=grid))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER_SCAN1)(VERSION=186647552)) * status * 0
29-FEB-2024 10:13:51 * version * 0
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.1.1)(PORT=1521)))
29-FEB-2024 10:13:52 * service_register * LsnrAgt * 0
29-FEB-2024 10:13:52 * service_register * lxxxxxxgl1 * 0
............
29-FEB-2024 10:09:49 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=60072)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred
29-FEB-2024 10:09:49 * <unknown connect data> * (ADDRESS=(PROTOCOL=tcp)(HOST=192.1.1.1)(PORT=60074)) * establish * <unknown sid> * 12525
TNS-12525: TNS:listener has not received client's request in time allowed
 TNS-12535: TNS:operation timed out
  TNS-12606: TNS: Application timeout occurred

2、查看确认一下对应IP的高频率连接问题

情况监听日志,几分钟后的查看

grid@HAHA trace\]$ cat listener_scan1.log \|grep 192.1.1.1\|wc -l 19329 3、根据有问题的IP,找软件维护人员查看,关闭对应程序后,系统恢复。

相关推荐
Edingbrugh.南空1 小时前
Flink SQLServer CDC 环境配置与验证
数据库·sqlserver·flink
码不停蹄的玄黓1 小时前
MySQL分布式ID冲突详解:场景、原因与解决方案
数据库·分布式·mysql·id冲突
爱上语文2 小时前
Redis基础(6):SpringDataRedis
数据库·redis·后端
Java初学者小白2 小时前
秋招Day14 - Redis - 应用
java·数据库·redis·缓存
丶意冷3 小时前
mybatisPlus分页方言设置错误问题 mybatisPlus对于Oceanbase的Oracle租户分页识别错误
java·数据库·oracle·oceanbase
时序数据说5 小时前
为什么时序数据库IoTDB选择Java作为开发语言
java·大数据·开发语言·数据库·物联网·时序数据库·iotdb
戒不掉的伤怀5 小时前
【Navicat 连接MySQL时出现错误1251:客户端不支持服务器请求的身份验证协议;请考虑升级MySQL客户端】
服务器·数据库·mysql
cv高级工程师YKY5 小时前
服务器 - - QPS与TPS介绍
数据库
nbsaas-boot5 小时前
高可扩展属性建模设计:架构师的全局思考与落地方案
数据库
爱上语文6 小时前
Redis基础(5):Redis的Java客户端
java·开发语言·数据库·redis·后端