Oracle service failover后自动切回原来的node

After database service resources in CRS failed over to another node in a RAC Cluster, they were terminated and automatically fail back to the original node incorrectly.

CAUSE

The "fail back" was actually a relocation of the service resource, initiated by user process(UiServer):


crsd_148.trc:2017-12-19 12:24:32.701587 :UiServer:1239340800: {1:54916:896} Container [ Name: UI_RELOCATE

crsd_148.trc:2017-12-19 12:24:32.701630 :UiServer:1239340800: {1:54916:896} Sending to PE. ctx= 0x7f830003a9f0, ClientPID=96087 ===========>

crsd_148.trc:2017-12-19 12:24:32.707319 :UiServer:1239340800: {1:54916:896} Response: c4|5!ORDERk7|MESSAGEt79|CRS-2673: Attempting to stop 'ora.orcl.orclsrv.svc' on 'node1'k7|MSGTY

PEt1|3k5|OBJIDt33|ora.orcl.orclsrv.svc 1 1k4|WAITt1|0

crsd_148.trc:2017-12-19 12:24:32.777894 :UiServer:1239340800: {1:54916:896} Response: c4|5!ORDERk7|MESSAGEt78|CRS-2677: Stop of 'ora.orcl.orclsrv.svc' on 'node1' succeededk7|MSGTYP

Et1|3k5|OBJIDt33|ora.orcl.orclsrv.svc 1 1k4|WAITt1|0

crsd_148.trc:2017-12-19 12:24:32.780142 : AGFW:1251948288: {1:54916:896} Agfw Proxy Server received the message: RESOURCE_START[ora.orcl.orclsrv.svc 1 1] ID 4098:8698047

crsd_148.trc:2017-12-19 12:24:32.780238 : AGFW:1251948288: {1:54916:896} Creating the resource: ora.orcl.orclsrv.svc 1 1

crsd_148.trc:2017-12-19 12:24:32.780339 : AGFW:1251948288: {1:54916:896} Initializing the resource ora.orcl.orclsrv.svc 1 1 for type ora.service.type

crsd_148.trc:2017-12-19 12:24:32.780380 : AGFW:1251948288: {1:54916:896} SR: acl = owner:oracle:rwx,pgrp:dba:r--,other::r--,group:dba:r-x,user:oracle:r-x

crsd_148.trc:2017-12-19 12:24:32.780665 : AGFW:1251948288: {1:54916:896} Agfw Proxy Server sending message: RESOURCE_ADD[ora.orcl.orclsrv.svc 1 1] ID 4356:2828 to the agent /u01/app/12.1.0.2/grid/bin/oraagent_oracle

The relocate was initiated by a clientPID 96087, using a "JAVA" program:

2017-12-19 12:24:32.701587 :UiServer:1239340800: {1:54916:896} Container [ Name: UI_RELOCATE

API_HDR_VER:

TextMessage[3]

CLIENT:

TextMessage[]

CLIENT_NAME:
TextMessage[java] ==============>

CLIENT_PID:
TextMessage[96087] ================>

CLIENT_PRIMARY_GROUP:

TextMessage[dba]

Further analysis showed that this was caused due to the presence of a custom FAN (Fast Application Notification) script defined inside the $GRID_HOME/racg/usrco directory. The following example of the custom script shows that it forcibly relocates a service back to the preferred node after it has failed over to an available node:

Custom script:

if service is not running, then start it

echo " service stopped, starting" >> "$LOGFILE"

ORACLE_HOME/bin/srvctl start service -d "DATABASE" -s "service" \>\> "LOGFILE"

else

Service is running, but is it running on preferred instance?

RUNNING=( 'echo "$SRVSTATUS" | sed -rne "s/.* ([a-zA-Z0-9]+)/\1/p" | tr "," "\n"' )

echo "{RUNNING\[@\]} = {PREFERRED[@]}"

if ! in_array "INSTANCE" "{RUNNING[@]}" ; then

echo " not running on preferred INSTANCE" \>\> "LOGFILE"

Find the first non-preferred running instance

CURRENT=""

for inst in "${RUNNING[@]}"; do

if ! in_array "inst" "{PREFERRED[@]}" ; then

CURRENT="$inst"

break

fi

done

Relocate

if [[ -n "$CURRENT" ]]; then

echo " relocate CURRENT -\> INSTANCE" >> "LOGFILE" **ORACLE_HOME/bin/srvctl relocate service -d "DATABASE" -s "service" -i "CURRENT" -t "INSTANCE" >> "$LOGFILE" <<<<<<<<<<<<<<<<<<<<<** <<<

fi

else

Service is already running on preferred instance, no need to do anything

echo " running on preferred INSTANCE" \>\> "LOGFILE"

fi

fi

fi

fi

done

SOLUTION

  1. If the custom script is a requirement in the environment, ignore the relocation.

  2. If the custom script is not a requirement or not wanted, verify and remove any user defined scripts in the '$GRID_HOME/racg/usrco' directory that were created for the services fail back.

相关推荐
伤不起bb38 分钟前
Redis 哨兵模式
数据库·redis·缓存
卑微的Coder40 分钟前
Redis Set集合命令、内部编码及应用场景(详细)
java·数据库·redis
2501_9153738840 分钟前
Redis线程安全深度解析:单线程模型的并发智慧
数据库·redis·安全
呼拉拉呼拉43 分钟前
Redis知识体系
数据库·redis·缓存·知识体系
霖檬ing44 分钟前
Redis——主从&哨兵配置
数据库·redis·缓存
卜及中4 小时前
【Redis/2】核心特性、应用场景与安装配置
数据库·redis·缓存
LucianaiB4 小时前
如何做好一份优秀的技术文档:专业指南与最佳实践
android·java·数据库
Eiceblue4 小时前
Python读取PDF:文本、图片与文档属性
数据库·python·pdf
敖云岚7 小时前
【Redis】分布式锁的介绍与演进之路
数据库·redis·分布式
LUCIAZZZ8 小时前
HikariCP数据库连接池原理解析
java·jvm·数据库·spring·springboot·线程池·连接池