Today, we had an issue where the asmlib disk label was removed from a disk at some point in the past. When we added new storage and issued
oracleasm scan disks
, the disk was “cleaned”, rendering it unavailable on the server from which the scan command was run. The data itself was OK, as evidenced by the fact the database was fully available on the other two nodes in the cluster.
We shutdown the instance on the affected node and began to troubleshoot. Using kfed, we found that the disk header itself was OK, but the provstr variable was set to ORCLDISK, whereas it should have been ORCLDISKDATA001, or whatever it had originally been labelled by asmlib. Using kfed, we found the structure of the disk header, but couldn’t figure out the actual binary structure to see how long the provstr was in terms of allocated storage. Using what is below, it appears to be 34 bytes. We base this on the fact that bytes 40 through 65 are null after the ORCLDISK piece, starting at byte 32.
14:05:20 oracle@oradb02dxdu ~ >cat asmmeta.py
#!/usr/bin/env python
f = open("/dev/oracleasm/disks/G11_ORADEV_DATA004", "rb")
try:
byte = f.read(1)
i = 1
while i <= 100:
byte = f.read(1)
print i,byte
i = i + 1
finally:
f.close()
14:05:33 oracle@oradb02dxdu ~ >./asmmeta.py
1 �
2 _
3 _
4
5
6
7
8 _
9
10
11 �
12
13 �
14 _
15 T
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32 O
33 R
34 C
35 L
36 D
37 I
38 S
39 K
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66 _
67
68 _
69
70 _
71 _
72 L
73 X
74 R
75 A
76 C
77 _
78 A
79 S
80 M
81 _
82 S
83 E
84 E
85 D
86 _
87 0
88 0
89 0
90 4
91
92
93
94
95
96
97
98
99
100
14:05:36 oracle@oradb02dxdu ~ >
Based on this, and consultation with Oracle support, we elected to:
* shutdown the database
* unmount the affected diskgroup
* rename the disk to what it should have been
* remount the diskgroup
* restart the database
Oracle suggested that we not drop the disk, label it, and re-add it. Since it was part of a cluster, and some of the metadata would be cached, they said this may prevent a drop, and possibly impact something else, as well.
We still don’t know how it was originally “unlabelled”.