At times I would like to know how a file is stored in HDFS. What is below will show which blocks exist for a given file, as well as on which nodes they are stored. import java.io.*; import java.util.*; import…
Category: Hadoop
Does hadoop/HDFS distribute writes to all data nodes on ingest?
I like simple, command line test cases. Lather, rinse, repeat (do any shampoo bottles actually have that anymore 🙂 ?) I wanted to ensure I could prove that ingests to hadoop actually didn’t send everything through the name node, which…
HBase JMX metrics
HBase exposes several metrics via JMX beans, some of which are similar to the Oracle performance counters recorded by AWR. Actually, they aren’t even *close* to what Oracle provides, but one can hope 🙂 Below is a simple example of…
Printing hadoop properties
The hadoop Configuration class implements the Iterable interface, so you can simply create a default configuration and list all properties, or pass a custom XML configuration file and print the properties from that. Below is a simple example. import java.util.*;…
Getting started with map reduce
Map Reduce is a computer technology algorithm that is built on the age old concept of functional programming. The goals of functional programming can be simplistically described as: every program returns a value “map” lists of values to functions Map…