This assumes the tables have no integrity constraints related to a table in another schema. This just loops until all tables are dropped. If you are the nervous type, add an exception handler for the integrity constraint exception, and exit…
Hive transactions
Below is just a simple example of hive transactions. These are very useful on slowly changing type 1 dimension tables for which you do not wish to retain history, but only the most recent value of the row. The table…
JDBC batch re-processing
I have always wanted to duplicate how GoldenGate handles batch failures. In GoldenGate, you can set the parameter “batchsql BATCHTRANSOPS 2000”, for example, to send 2000 statements at a time to the database for execution. In general, this is far…
zookeeper and reverse lookups
Suffice it to say, on zookeeper 3.4.6 (at least) you will have all kinds of grief in a kerberos secured cluster if the zookeeper nodes can’t be looked up by passing the IP to get the hostname. Specifically, in the…
Reading random lines with python
Technically, this isn’t random, but it met my needs. I wanted to read an arbitrary number of lines from a 4GB text file to spot check data we had loaded. What is below does the following: 1) get the file…
JAAS with Client/Server socket example
What is below is a hack on the same example floating all around the internet for using JAAS with Kerberos (Active Directory, in this case). This extends the example to send the ticket over a network socket to the server.…