<\/p>\nIn earlier versions of hadoop, the namenode was the Achilles heel. While there was the option of failing over to a secondary namenode, this required manual intervention, or heavy scripting at best. Even then, failover wasn’t instantaneous.<\/p>\n
With Hadoop 0.23 and later releases, a primary and standby namenode configuration was introduced. This required very little out of the box configuration, with tools such as ambari providing a next\/next\/next click setup.<\/p>\n
This post simply shows how it works, with an example that shows each node in the configuration. We then start a job that will require access to the primary namenode, and terminate its process. We see that our job runs to completion.<\/p>\n
First, we show our primary namenode..<\/p>\n
<\/p>\n
…and then, our secondary namenode…<\/p>\n
<\/p>\n
We then compile our simple class shown below…<\/p>\n
\r\nimport java.io.*;\r\nimport java.util.*;\r\nimport org.apache.hadoop.conf.*;\r\nimport org.apache.hadoop.fs.*;\r\n\r\npublic class HDFSFileBlocks {\r\n public static void main(String[] args) throws IOException {\r\n Configuration conf = new Configuration();\r\n\r\n FileSystem fs = FileSystem.get(conf);\r\n Path file = new Path(args[0]);\r\n FileStatus fileStatus = fs.getFileStatus(file);\r\n BlockLocation[] blocks = fs.getFileBlockLocations(fileStatus, 0, fileStatus.getLen());\r\n for (int i = 0; i < blocks.length; i++) {\r\n System.out.println(blocks[i].toString());\r\n }\r\n BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(file)));\r\n String line=br.readLine();\r\n while (line != null){\r\n System.out.println(line);\r\n line=br.readLine();\r\n }\r\n fs.close();\r\n }\r\n}\r\n<\/pre>\n...and run the job, during which time we terminate the primary namenode on node 14...<\/p>\n
\r\n[root@cmhl******07 ~]# export CLASSPATH=$(yarn classpath)\r\n[root@cmhl******07 ~]# javac HDFSFileBlocks.java\r\n[root@cmhl******07 ~]# cat manifest.txt\r\nManifest-Version: 1.0\r\nMain-Class: HDFSFileBlocks\r\n\r\n[root@cmhl******07 ~]# jar cfm myjar.jar manifest.txt HDFSFileBlocks.class\r\n[root@cmhl******07 ~]# hadoop jar myjar.jar \/tmp\/splunk__106646.esw3c_S.201311290900-1000-0 > \/dev\/null & ssh cmhl*******14 \"kill -9 \\$(ps -ef | grep java | grep -v grep | egrep NameNode\\$ | awk '{print \\$2}')\"[4] 13071[3]\u00a0\u00a0 Done\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop jar myjar.jar \/tmp\/splunk__106646.esw3c_S.201311290900-1000-0 > \/dev\/null\r\n[root@cmhl******07 ~]#\r\n<\/pre>\nWe see that our standby namenode is stopped...<\/p>\n
<\/p>\n
...and also see the primary namenode is now on node 13...<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
In earlier versions of hadoop, the namenode was the Achilles heel. While there was the option of failing over to a secondary namenode, this required manual intervention, or heavy scripting at best. Even then, failover wasn’t instantaneous. With Hadoop 0.23…<\/p>\n