After restarting a test server, I found the RAC VIP would not start. This post is just a quick note as to how I troubleshot the problem, as well as its ultimate resolution.
I frequently use the
find -mmin 5 -type f | xargs grep -i "whatever you want"
command to find any recently modified files under a directory where I believe a logfile may live. In this case, that proved to be useful. I cd’d to the GRID_INFRASTRUCTURE home, and found references to VIP errors in the following file:
$GRID_HOME/log/$(hostname)/agent/crsd/orarootagent_root/orarootagent_root.log
In this file, I found the following contents…
2011-11-01 20:26:31.761: [ AGFW][2765908880] {1:2236:4470} Agent received the message: RESOURCE_START[ora.dell11gr1.vip 1 1] ID 4098:29951
2011-11-01 20:26:31.761: [ AGFW][2765908880] {1:2236:4470} Preparing START command for: ora.dell11gr1.vip 1 1
2011-11-01 20:26:31.761: [ AGFW][2765908880] {1:2236:4470} ora.dell11gr1.vip 1 1 state changed from: OFFLINE to: STARTING
2011-11-01 20:26:31.763: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] clsn_agent::start {
2011-11-01 20:26:31.781: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] InterfaceName = eth0
2011-11-01 20:26:31.781: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] NetworkMask = 255.255.255.0
2011-11-01 20:26:31.782: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] NetworkAuto = static
2011-11-01 20:26:31.782: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] Checking if eth0 Interface is fine
2011-11-01 20:26:31.791: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] ifname=eth0
2011-11-01 20:26:31.791: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] subnetmask=255.255.255.0
2011-11-01 20:26:31.791: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] subnetnumber=192.168.1.0
2011-11-01 20:26:31.792: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] InterfaceName = eth0
2011-11-01 20:26:31.970: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] HostName dell11gr1-vip translated to
2011-11-01 20:26:31.970: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] Interface Name = eth0
2011-11-01 20:26:31.970: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] Ip Address =
2011-11-01 20:26:31.971: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] clsn_agent::start: Exception AgentException
2011-11-01 20:26:31.971: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] InterfaceName = eth0 IpAddress =
2011-11-01 20:26:31.971: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [start] clsn_agent::start }
2011-11-01 20:26:31.971: [ AGFW][2900142992] {1:2236:4470} Command: start for resource: ora.dell11gr1.vip 1 1 completed with status: FAIL
2011-11-01 20:26:31.972: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] Waiting for Network Object to be initialized
2011-11-01 20:26:31.973: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] InterfaceName = eth0
2011-11-01 20:26:31.973: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] NetworkMask = 255.255.255.0
2011-11-01 20:26:31.973: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] NetworkAuto = static
2011-11-01 20:26:31.973: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] Checking if eth0 Interface is fine
2011-11-01 20:26:31.978: [ AGFW][2765908880] {1:2236:4470} Agent sending reply for: RESOURCE_START[ora.dell11gr1.vip 1 1] ID 4098:29951
2011-11-01 20:26:32.008: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] ifname=eth0
2011-11-01 20:26:32.008: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] subnetmask=255.255.255.0
2011-11-01 20:26:32.008: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] subnetnumber=192.168.1.0
2011-11-01 20:26:32.008: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] InterfaceName = eth0
2011-11-01 20:26:32.008: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] HostName dell11gr1-vip translated to
2011-11-01 20:26:32.009: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] Interface Name = eth0
2011-11-01 20:26:32.009: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] Ip Address =
2011-11-01 20:26:32.009: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] AgentException caught in VipAgent::check()
2011-11-01 20:26:32.009: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] InterfaceName = eth0 IpAddress =
2011-11-01 20:26:32.010: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] VipAgent::sendFail2Srv {
2011-11-01 20:26:32.033: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] VipAgent::sendFail2Srv }
Notice the following line in particular…
2011-11-01 20:26:32.008: [ora.dell11gr1.vip][2900142992] {1:2236:4470} [check] HostName dell11gr1-vip translated to
I then found that I couldn’t ping the VIP by name…
dell11gr1:oracle:+ASM1:/u01/app/11.2.0/grid# ping dell11gr1-vip
ping: unknown host dell11gr1-vip
The solution in this case was simple, I just needed to start my DNS test server.