We had a need to authenticate user requests against AD in a kerberos enabled cluster, and allow “local” hive sessions to use only a keytab. Below are the examples of each. First, we show how to connect over a binary…
Category: Database
Accessing Hive via DBVisualizer
Download the most recent free version of DB Visualizer from the following URL: http://www.dbvis.com/ Execute the downloaded file. If you see the following, it means you have either not installed java, or the installer can’t find where you installed it.…
Corrupt index not detected by analyze table validate structure cascade
Pretty wild stuff. This was in 11.2.0.3.9, EE… SQL> select * from dcspp_order_rel where order_id = ‘EXP1439962334’ and sequence_num = 5; ORDER_ID RELATIONSHIPS SEQUENCE_NUM —————————————- —————————————- ———— EXP1439962334 r406622317 5 SQL> select * from dcspp_order_rel where order_id = ‘EXP1439962334’ and…
Are files in HDFS immutable?
Call me cynical, I just am a bit of a doubting Thomas. Using our previous write test code, we simply run the exact same test, only we do it twice. [root@cmhlpdlkedat15 ~]# hadoop HDFSWriteTest foobar.txt hdfs://cmhlpdlkedat14.expressco.com:8020 [root@cmhlpdlkedat15 ~]# hdfs dfs…
Getting additional data on a slow hive reducer
We have a query that has been running for almost two hours on the last of 1,009 reducers in the first reduce phase. We wanted to gain additional insight into what it was doing, so we first find the node…
Drop all tables in a schema
This assumes the tables have no integrity constraints related to a table in another schema. This just loops until all tables are dropped. If you are the nervous type, add an exception handler for the integrity constraint exception, and exit…
Hive transactions
Below is just a simple example of hive transactions. These are very useful on slowly changing type 1 dimension tables for which you do not wish to retain history, but only the most recent value of the row. The table…
JDBC batch re-processing
I have always wanted to duplicate how GoldenGate handles batch failures. In GoldenGate, you can set the parameter “batchsql BATCHTRANSOPS 2000”, for example, to send 2000 statements at a time to the database for execution. In general, this is far…
Hadoop HA namenode example
In earlier versions of hadoop, the namenode was the Achilles heel. While there was the option of failing over to a secondary namenode, this required manual intervention, or heavy scripting at best. Even then, failover wasn’t instantaneous. With Hadoop 0.23…
PURGE DBA_RECYCLEBIN taking a long time
Just check DBA_SEGMENTS for SEGMENT_NAME like ‘BIN%’, and it should be dropping in number… SQL> select count(*) from dba_segments where segment_name like ‘BIN%’; COUNT(*) ———- 2307 SQL> select count(*) from dba_segments where segment_name like ‘BIN%’; COUNT(*) ———- 2297 SQL> select…