{"id":4353,"date":"2014-10-24T17:08:27","date_gmt":"2014-10-24T22:08:27","guid":{"rendered":"http:\/\/appcrawler.com\/wordpress\/?p=4353"},"modified":"2014-10-24T17:20:42","modified_gmt":"2014-10-24T22:20:42","slug":"oracle-database-on-nfs-failure-scenario-testing","status":"publish","type":"post","link":"http:\/\/appcrawler.com\/wordpress\/2014\/10\/24\/oracle-database-on-nfs-failure-scenario-testing\/","title":{"rendered":"Oracle database on NFS – Failure scenario testing"},"content":{"rendered":"

This post will provide a simple test case for the efficacy of creating and using an Oracle database cluster using NFS storage. The basic configuration will be shown, as well as the test case scenarios and results.\u00a0 A complete installation of an Oracle RAC environment will not be shown.<\/p>\n

From a high level, the installation and configuration requires only the following:<\/p>\n

Create NFS server
\nCreate and export directory on NFS server
\nCreate two Linux servers
\nCreate cluster on linux servers using NFS filesystem for storage
\nCreate database on cluster using NFS filesystem for storage<\/p>\n

CONFIGURATION<\/p>\n

All environments are virtualized, and consist of the following components:<\/p>\n

Windows Server 2008 for Active Directory and DNS services
\nCentos 6.5 (2.6.32-431.el6.x86_64 kernel) for the following guests:
\n* rac01
\n* rac02
\n* nfs01<\/p>\n

On each of the Linux guests, there is a separate network interface for the public IP, the private cluster communication IP, and the storage IP.\u00a0 This more closely resembles production and allows us to test the failure of individual components.<\/p>\n

Below is the output of our base installation:<\/p>\n

\r\n  rac01:oracle:nfsdb:\/home\/oracle>.\/crsstat.sh\r\n  NAME\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 TARGET\u00a0\u00a0\u00a0\u00a0 STATE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 SERVER\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 STATE_DETAILS\r\n  ------------------------- ---------- ----------\u00a0\u00a0\u00a0\u00a0\u00a0 ------------ ------------------\r\n  ora.LISTENER.lsnr\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\r\n  ora.LISTENER.lsnr\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.asm\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Instance Shutdown\r\n  ora.asm\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.gsd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\r\n  ora.gsd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0 OFFLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.net1.network\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\r\n  ora.net1.network\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.ons\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\r\n  ora.ons\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.LISTENER_SCAN1.lsnr\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0rac02\r\n  ora.cvu\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.nfsdb.db\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Open\r\n  ora.nfsdb.db\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Open\r\n  ora.oc4j\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\r\n  ora.rac01.vip\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac01\r\n  ora.rac02.vip\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  ora.scan1.vip\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0 ONLINE\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rac02\r\n  rac01:oracle:nfsdb:\/home\/oracle>\r\n<\/pre>\n
We show where the cluster related files are stored\u2026<\/p>\n
\r\n  rac01:oracle:nfsdb:\/u01\/oradata\/storage>ls  -lrt\r\n  total  23540\r\n  -rw-r-----.  1 root dba 272756736 Sep\u00a0 2 09:55 ocr\r\n  -rw-r-----.  1 grid dba\u00a0 21004800 Sep\u00a0 2 10:00 vdsk\r\n<\/pre>\n
We then show the database file locations\u2026<\/p>\n
\r\n  rac01:oracle:nfsdb:\/u01\/oradata\/db>ls -lrt\r\n  total 8\r\n  drwxr-x---.  5 oracle dba 4096 Sep\u00a0 2 09:00 NFSDB  \r\n  drwxr-x---.  2 oracle dba 4096 Sep\u00a0 2 09:53 nfsdb\r\n  rac01:oracle:nfsdb:\/u01\/oradata\/db>ls -lrt NFSDB\/datafile\/\r\n  total 3022556\r\n  -rw-r-----.  1 oracle dba\u00a0 30416896 Sep\u00a0 2 09:09 o1_mf_temp_b0chx0oq_.tmp\r\n  -rw-r-----.  1 oracle dba 545267712 Sep\u00a0 2 09:10 o1_mf_sysaux_b0chqslw_.dbf\r\n  -rw-r-----.  1 oracle dba\u00a0\u00a0 5251072 Sep\u00a0 2 09:10 o1_mf_users_b0chqsrt_.dbf\r\n  -rw-r-----.  1 oracle dba 328343552 Sep\u00a0 2 09:10 o1_mf_example_b0chxg1g_.dbf\r\n  -rw-r-----.  1 oracle dba\u00a0 26222592 Sep\u00a0 2 09:10 o1_mf_undotbs2_b0cj8z4d_.dbf\r\n  -rw-r-----.  1 oracle dba 104865792 Sep\u00a0 2 09:10 o1_mf_undotbs1_b0chqsoo_.dbf\r\n  -rw-r-----.  1 oracle dba 754982912 Sep\u00a0 2 09:10 o1_mf_system_b0chqsc3_.dbf\r\n  -rw-r-----.  1 oracle dba\u00a0 20979712 Sep \u00a02 09:53 o1_mf_temp_b0clwoqd_.tmp\r\n  -rw-r-----.  1 oracle dba\u00a0\u00a0 5251072 Sep\u00a0 2 09:54 o1_mf_users_b0clqtm9_.dbf\r\n  -rw-r-----.  1 oracle dba\u00a0 26222592 Sep\u00a0 2 10:00 o1_mf_undotbs2_b0clxo1j_.dbf\r\n  -rw-r-----.  1 oracle dba 524296192 Sep\u00a0 2 10:00 o1_mf_sysaux_b0clqtgb_.dbf\r\n  -rw-r-----.  1 oracle dba\u00a0 36708352 Sep\u00a0 2 10:00 o1_mf_undotbs1_b0clqtjh_.dbf\r\n  -rw-r-----.  1 oracle dba 734011392 Sep\u00a0 2 10:00 o1_mf_system_b0clqt70_.dbf\r\n  rac01:oracle:nfsdb:\/u01\/oradata\/db>\r\n<\/pre>\n
  \u00a0
\n\u2026and finally, the state of each instance of the database\u2026<\/p>\n
  \u00a0\r\n  SQL> select host_name,status from gv$instance;\r\n  \u00a0\r\n  HOST_NAME\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 STATUS\r\n  -------------------- ------------\r\n  rac01.howard.local\u00a0\u00a0 OPEN\r\n  rac02.howard.local\u00a0\u00a0 OPEN\r\n  \u00a0\r\n  SQL>\r\n<\/pre>\n
TESTING<\/b><\/p>\n
Oracle failure algorithm<\/p>\n
Before describing our test scenarios, some background on how different failures are handled by the Oracle software would be beneficial.<\/p>\n
Oracle handles failures based on a two prong approach.<\/p>\n
1) Network communication \u2013 This is checked once per second by each node in the cluster to ensure it can successfully reach the others using the private cluster connection
\n2) Voting disk checks \u2013 Once per second, the oracle software issues a pwrite() call to the cluster voting disk with a supplied offset 512 bytes in size.\u00a0 Each node in the cluster has its own 512 byte \u201cslot\u201d in the voting disk file for status<\/p>\n
The failure can be handled in an orderly manner if either of the above components and associated checks fail on a given server, or even multiple servers.\u00a0 This is defined as follows:<\/p>\n
*  Voting disk access failures will be allowed 200 seconds to self-heal, as long as the network heartbeats between nodes in the cluster are successful
\n*  Network heartbeat failures will be allowed 30 seconds to self-heal, as long as the failing node(s) can continue to write their status to the shared voting disk for the other nodes to see<\/p>\n
After each time period above, if the respective failure condition is still in place, the failing node will \u201ccommit suicide\u201d by taking itself out of the cluster.\u00a0 Oracle 11.2.0 introduced fencing that did not always require a node reboot.\u00a0 The Oracle High Availability service (\u201cOHAS\u201d) will be used to trigger actions in the clusterware that result in nodes removing and adding themselves from and to the active list.<\/p>\n
Test harness<\/p>\n
Stun various servers by pausing guest in VirtualBox manager
\nRun java command line program that threads a connection to each instance in the cluster.\u00a0 This will be used to exercise the database as well as print the results of that activity.\u00a0 This is included as APPENDIX A<\/p>\n
Specific tests<\/p>\n
Single server has a failed connection to the NFS server<\/b><\/p>\n
When we run our test and fail the storage connection for node 2, notice that no activity occurs for 207 seconds after the previous successful update.\u00a0 This is due to the fact the row is locked by the failed node, and this is not resolved until the node has been evicted.\u00a0 Notice also that this time is comprised of the 200 second timeout provided for any possible resolution of disk access failures, plus a few seconds for the actual eviction.<\/p>\n
  \u00a0\r\n  Tue Sep 02 12:31:15 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 12:31:16 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 12:31:16 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 12:31:17 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 12:31:17 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 12:31:18 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  java.sql.SQLException:  No more data to read from socket\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1199)\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:308)  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \r\n          at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:199)\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:542)\r\n\r\n  \r\n\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 at testIt.run(testIt.java:32)\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0at java.lang.Thread.run(Thread.java:637)\r\n  Tue Sep 02 12:34:45 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 12:34:46 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 12:34:47 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 12:34:48 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 12:34:49 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n<\/pre>\n
NFS server pause<\/b>
\n\u00a0
\nIn this test, we simulate a high availability event in which the enterprise NFS server stack fails a component of its environment.\u00a0To simulate this, we pause the NFS server in the VirtualBoxManager UI for about ten seconds.<\/p>\n
As we can see, the software was blocked for about 13 seconds, but no cluster reconfiguration occurred.<\/p>\n
  \u00a0\r\n  rac01:oracle:nfsdb1:\/home\/oracle>java testIt\r\n  Tue Sep 02 13:05:20 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 13:05:20 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 13:05:21 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 13:05:21 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 13:05:22 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 13:05:22 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 13:05:23 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 13:05:23 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 13:05:24 GMT-05:00 2014\u00a0\u00a0\u00a0 \u00a0\u00a0updated table in thread 1  \r\n  Tue Sep 02 13:05:24 GMT-05:00 2014\u00a0\u00a0 \u00a0\u00a0\u00a0updated table in thread 2  \r\n  Tue Sep 02 13:05:37 GMT-05:00 2014\u00a0 \u00a0\u00a0\u00a0\u00a0updated table in thread 1\r\n  Tue Sep 02 13:05:37 GMT-05:00 2014 \u00a0\u00a0\u00a0\u00a0 updated table in thread 2  \r\n  Tue Sep 02 13:05:38 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1  \r\n  Tue Sep 02 13:05:38 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 13:05:39 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 13:05:39 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n  Tue Sep 02 13:05:40 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 1\r\n  Tue Sep 02 13:05:40 GMT-05:00 2014\u00a0\u00a0\u00a0\u00a0\u00a0 updated table in thread 2\r\n<\/pre>\n
  \u00a0
\nNFS server pause when the database storage is accessed separately from the cluster devices<\/b><\/p>\n
This is not provided, as it is expected the storage will be accessed over a common network interface.\u00a0If necessary, it would be trivial to add this to the test list.\u00a0 All that would be required is a separate interface for the database storage, as well as a separately mounted filesystem.<\/p>\n
APPENDIX A<\/p>\n
\r\n  import java.sql.*;\r\n  import java.util.*;\r\n  \u00a0\r\n  public class testIt implements Runnable {\r\n  \u00a0 String server;\r\n  \u00a0 Thread t;\r\n  \u00a0 static Random r;\r\n  \u00a0 public static void main (String args[]) {\r\n  \u00a0\u00a0\u00a0 r = new Random();\r\n  \u00a0\u00a0\u00a0 testIt t1 = new testIt(\"1\");\r\n  \u00a0\u00a0 \u00a0testIt t2 = new testIt(\"2\");\r\n  \u00a0 }\r\n  \u00a0\r\n  \u00a0 testIt(String server) {\r\n  \u00a0\u00a0\u00a0 try {\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 t = new Thread(this);\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 this.server = server;\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 t.start();\r\n  \u00a0\u00a0\u00a0 }\r\n  \u00a0\u00a0\u00a0 catch (Exception e) {\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 e.printStackTrace();\r\n  \u00a0\u00a0\u00a0 }\r\n  \u00a0 }\r\n  \u00a0\r\n  \u00a0 public void run () {\r\n  \u00a0\u00a0\u00a0 try {\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 Class.forName(\"oracle.jdbc.driver.OracleDriver\");\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 Connection conn = DriverManager.getConnection(\"jdbc:oracle:thin:system\/welcome@rac0\"  +  server + \":1521:nfsdb\" + server);\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 PreparedStatement pst = conn.prepareStatement(\"update test set c = ?\");\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 while (true) {\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pst.setInt(1,r.nextInt());\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pst.execute();\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 System.out.println(new java.util.Date().toString() + '\\t' + \"updated table in thread \" + this.server);\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Thread.sleep(1000);\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n  \u00a0\u00a0\u00a0 }\r\n  \u00a0\u00a0\u00a0 catch (Exception e) {\r\n  \u00a0\u00a0\u00a0\u00a0\u00a0 e.printStackTrace();\r\n  \u00a0\u00a0\u00a0 }\r\n  \u00a0 }\r\n  }\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
This post will provide a simple test case for the efficacy of creating and using an Oracle database cluster using NFS storage. The basic configuration will be shown, as well as the test case scenarios and results.\u00a0 A complete installation…<\/p>\n
Read more →<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[19,22,32],"tags":[],"_links":{"self":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/4353"}],"collection":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/comments?post=4353"}],"version-history":[{"count":15,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/4353\/revisions"}],"predecessor-version":[{"id":4506,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/4353\/revisions\/4506"}],"wp:attachment":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/media?parent=4353"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/categories?post=4353"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/tags?post=4353"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}