{"id":2298,"date":"2012-06-14T13:17:09","date_gmt":"2012-06-14T18:17:09","guid":{"rendered":"http:\/\/appcrawler.com\/wordpress\/?p=2298"},"modified":"2012-06-14T13:18:00","modified_gmt":"2012-06-14T18:18:00","slug":"instance-not-restarting-after-failure-in-11-2-0-3","status":"publish","type":"post","link":"http:\/\/appcrawler.com\/wordpress\/2012\/06\/14\/instance-not-restarting-after-failure-in-11-2-0-3\/","title":{"rendered":"Instance not restarting after failure in 11.2.0.3"},"content":{"rendered":"<p>I thought this was interesting.  I always assumed that the RESTART_ATTEMPTS parameter of a cluster resource was only incremented after a failure to restart it.  However, after developing an internal training course for our team, I noticed that after a couple of induced failures and the clusterware <\/b>successfully<\/b> restarting it, on a third induced failure the clusterware wouldn&#8217;t restart it.  We found what is below in the $GRID_HOME\/log\/$(hostname)\/agent\/crsd\/oraagent_oracle\/oraagent_oracle.log file&#8230;<\/p>\n<pre lang=\"text\">\r\n2012-06-14 13:28:38.129: [ USRTHRD][3959420672] {0:9:6} ClusterSubscriber::SubscriberWorker::InternalClusterSubscriber::handleEventCBexecuting for reason 1\r\n2012-06-14 13:28:38.129: [ USRTHRD][3959420672] {0:9:6} event type is CRS_NOT_RESTARTING\r\n2012-06-14 13:28:38.129: [ USRTHRD][3959420672] {0:9:6} bodylen = 528\r\n2012-06-14 13:28:38.129: [ USRTHRD][3959420672] {0:9:6} -----------BodyBlock----------\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  ACTION='1'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  CLS_TINT='{0:9:6}'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  CURRENT_STATE='OFFLINE'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  DATABASE_TYPE='RAC'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  DB_UNIQUE_NAME='express.home'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  INSTANCE_NAME='express1'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  NAME='ora.express.db'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  NUMBER_OF_ATTEMPTS='2'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  REASON='NOMORE_RESTART_ATTEMPTS'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  RESOURCE_CLASS='database'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  RESOURCE_INCARNATION_NUMBER='4'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  RESOURCE_LOCATION='expressdb1'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  SEQUENCE_NUMBER='300118'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  TARGET_STATE='ONLINE'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  TIMESTAMP='2012-06-14 13:28:38'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  TYPE='ora.database.type'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  USER='SYSTEM'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  Version='11.2.0.3.0'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  CLUSTER_NAME='expresscrs'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  DB_UNIQUE_NAME='express.home'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  ORACLE_CLUSTERWARE.SUBCOMPONENT='CRSD'\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  RESOURCE_CLASS='database'\r\n<\/pre>\n<p>Notice the following line&#8230;<\/p>\n<pre lang=\"text\">\r\n2012-06-14 13:28:38.130: [ USRTHRD][3959420672] {0:9:6}  REASON='NOMORE_RESTART_ATTEMPTS'\r\n<\/pre>\n<p>After we modified the resource, it successfully restarted it (up to five times).<\/p>\n<pre lang=\"text\">\r\nexpressdb1:grid:+ASM1:\/home\/grid# crsctl modify resource ora.express.db -attr RESTART_ATTEMPTS=5\r\nexpressdb1:grid:+ASM1:\/home\/grid# crs_stat -p ora.express.db\r\nNAME=ora.express.db\r\nTYPE=ora.database.type\r\nACTION_SCRIPT=\r\nACTIVE_PLACEMENT=1\r\nAUTO_START=restore\r\nCHECK_INTERVAL=1\r\nDESCRIPTION=Oracle Database resource\r\nFAILOVER_DELAY=0\r\nFAILURE_INTERVAL=60\r\nFAILURE_THRESHOLD=1\r\nGEN_START_OPTIONS@SERVERNAME(expressdb1)=open\r\nGEN_START_OPTIONS@SERVERNAME(expressdb2)=open\r\nGEN_USR_ORA_INST_NAME@SERVERNAME(expressdb1)=express1\r\nGEN_USR_ORA_INST_NAME@SERVERNAME(expressdb2)=express2\r\nHOSTING_MEMBERS=\r\nPLACEMENT=restricted\r\nRESTART_ATTEMPTS=5\r\nSCRIPT_TIMEOUT=60\r\nSTART_TIMEOUT=600\r\nSTOP_TIMEOUT=600\r\nUPTIME_THRESHOLD=1h\r\nUSR_ORA_INST_NAME@SERVERNAME(expressdb1)=express1\r\nUSR_ORA_INST_NAME@SERVERNAME(expressdb2)=express2\r\n\r\nexpressdb1:grid:+ASM1:\/home\/grid#\r\n<\/pre>\n<p>The RESTART_ATTEMPTS parameter must apply to any restart performed by the clusterware.  As soon as it is manually started, the restart counter is reset to 0.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I thought this was interesting. I always assumed that the RESTART_ATTEMPTS parameter of a cluster resource was only incremented after a failure to restart it. However, after developing an internal training course for our team, I noticed that after a&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"more-link\" href=\"http:\/\/appcrawler.com\/wordpress\/2012\/06\/14\/instance-not-restarting-after-failure-in-11-2-0-3\/\">Read more &rarr;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[19,22,32],"tags":[],"_links":{"self":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/2298"}],"collection":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2298"}],"version-history":[{"count":6,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/2298\/revisions"}],"predecessor-version":[{"id":2304,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/2298\/revisions\/2304"}],"wp:attachment":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2298"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2298"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2298"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}