I like simple, command line test cases. Lather, rinse, repeat (do any shampoo bottles actually have that anymore \ud83d\ude42 ?)<\/p>\n
I wanted to ensure I could prove that ingests to hadoop actually didn’t send everything through the name node, which would be a bottleneck.<\/p>\n
I proved that it does in fact, distribute writes over all nodes by the following test case using hadoop-0.23.7.<\/p>\n
From a client node (not a data node, or any part of the cluster) with a hadoop installation…<\/p>\n
Set up your CLASSPATH…<\/p>\n
\r\n#!\/bin\/sh\r\n\r\nexport CLASSPATH=.\r\n\r\nfor f in $HADOOP_HOME\/share\/hadoop\/common\/lib\/*; do export CLASSPATH=$CLASSPATH:${f}; done\r\nfor f in $HADOOP_HOME\/share\/hadoop\/common\/*; do export CLASSPATH=$CLASSPATH:${f}; done\r\nfor f in $HADOOP_HOME\/share\/hadoop\/hdfs\/*; do export CLASSPATH=$CLASSPATH:${f}; done\r\n\r\n<\/pre>\nCompile the following java class…<\/p>\n
\r\nimport java.io.*;\r\nimport java.util.*;\r\nimport org.apache.hadoop.conf.*;\r\nimport org.apache.hadoop.fs.*;\r\n\r\npublic class HDFSWriteTest {\r\n public static void main(String[] args) throws IOException {\r\n Configuration conf = new Configuration();\r\n FileSystem fs = FileSystem.get(conf);\r\n System.out.println(fs.getUri());\r\n Path file = new Path(args[0]);\r\n \/\/set file block size to 1048576 (1MB) to ensure the writes are more likely to go multiple nodes\r\n FSDataOutputStream outStream = fs.create(file,false, 4096, (short)3, (long)1048576);\r\n FileInputStream fstream = new FileInputStream(\"vals.txt\");\r\n DataInputStream in = new DataInputStream(fstream);\r\n BufferedReader br = new BufferedReader(new InputStreamReader(in));\r\n String strLine;\r\n while ((strLine = br.readLine()) != null) {\r\n outStream.writeUTF(strLine);\r\n }\r\n outStream.close();\r\n fs.close();\r\n }\r\n}\r\n<\/pre>\nLoad 800,000 rows into the source file we will load into hadoop\/HDFS…<\/p>\n
\r\n# for i in {1..800000}; do echo \"value number ${i}\" >> vals.txt ; done &\r\n<\/pre>\nTurn on tcpdump on the name node and at least one data node in the hadoop cluster…<\/p>\n
\r\n# tcpdump -A -s 1500 -nnn not port 22 > output.txt\r\n<\/pre>\nRun the java class…<\/p>\n
\r\n$HADOOP_HOME\/bin\/hadoop HDFSWriteTest foobar.txt\r\n<\/pre>\nCTRL-C the tcpdump commands on each host where they are running in the hadoop cluster, then analyze the output on each with the following. Change 192.168.3.100 to whatever your client machine (the one from which you ran the test) IP address is…<\/p>\n
\r\nawk '{if ($3 ~ \"192.168.3.100\") {l=1} else if (l == 1 && $0 ~ \"value number\") {s++;l=0}} END {print s}' output.txt\r\n<\/pre>\nYou should see traffic from the client machine to the respective datanodes on which you ran tcpdump.<\/p>\n","protected":false},"excerpt":{"rendered":"
I like simple, command line test cases. Lather, rinse, repeat (do any shampoo bottles actually have that anymore \ud83d\ude42 ?) I wanted to ensure I could prove that ingests to hadoop actually didn’t send everything through the name node, which…<\/p>\n