Today, I had an issue I had seen before, but I can never remember how I fix it. As such, I wanted to document it here.
Although rare, periodically an abnormally terminated instance will leave behind a shared memory segment. Since the SGA is attached by key which is the database name and instance_number, this means you can’t restart the instance until you remove this shared memory segment.
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 6160386 root 644 52 2
0x00000000 6193156 root 644 16384 2
0x00000000 6225925 root 644 268 2
0xea42bf0c 6881286 oracle 660 4096 0
0x00000000 7143431 oracle 660 608174080 1 dest
------ Semaphore Arrays --------
key semid owner perms nsems
0x3c490990 1310721 oracle 660 104
0xdd54209c 2228226 oracle 640 154
------ Message Queues --------
key msqid owner perms used-bytes messages
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>ipcrm -m 7143431
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>ipcs -p
------ Shared Memory Creator/Last-op --------
shmid owner cpid lpid
6160386 root 2382 2943
6193156 root 2382 2943
6225925 root 2382 2943
6881286 oracle 7223 31321
7143431 oracle 30892 31244
------ Message Queues PIDs --------
msqid owner lspid lrpid
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>ps -ef | grep 30892
oracle 31529 30326 0 21:40 pts/0 00:00:00 grep 30892
As you can see, the creating PID is no longer in the process table. Also, but not shown, the last process to attach to the shared memory segment had already been killed.
To resolve this, it’s lsof to the rescue…
lsof | grep shmid #in the ipcs output above.
Alternatively, you can brute force your way in if the attached PID is not a background process. See below…
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>ps -ef | grep LOCAL
oracle 30961 1 0 21:27 ? 00:00:04 oracle+ASM1_asmb_dublin (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 30985 1 0 21:27 ? 00:00:02 oracledublin (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 31547 30326 0 21:40 pts/0 00:00:00 grep LOCAL
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>kill -9 30961 30985
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 6160386 root 644 52 2
0x00000000 6193156 root 644 16384 2
0x00000000 6225925 root 644 268 2
0xea42bf0c 6881286 oracle 660 4096 0
------ Semaphore Arrays --------
key semid owner perms nsems
0x3c490990 1310721 oracle 660 104
0xdd54209c 2228226 oracle 640 154
------ Message Queues --------
key msqid owner perms used-bytes messages
dell11gr1:oracle:dublin:/u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin>