{"id":4725,"date":"2015-02-25T16:20:45","date_gmt":"2015-02-25T21:20:45","guid":{"rendered":"http:\/\/appcrawler.com\/wordpress\/?p=4725"},"modified":"2015-02-25T16:29:53","modified_gmt":"2015-02-25T21:29:53","slug":"zookeeper-and-reverse-lookups","status":"publish","type":"post","link":"http:\/\/appcrawler.com\/wordpress\/2015\/02\/25\/zookeeper-and-reverse-lookups\/","title":{"rendered":"zookeeper and reverse lookups"},"content":{"rendered":"<p>Suffice it to say, on zookeeper 3.4.6 (at least) you will have all kinds of grief in a kerberos secured cluster if the zookeeper nodes can&#8217;t be looked up by passing the IP to get the hostname.<\/p>\n<p>Specifically, in the \/var\/log\/krb5kdc.log file, you will see the following&#8230;<\/p>\n<pre>\r\nFeb 25 09:02:48 hdp.howard.local krb5kdc[1416](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 192.168.56.101: \r\nUNKNOWN_SERVER: authtime 0,  zookeeper\/hdp.howard.local@HADOOP.LOCAL for zookeeper\/192.168.56.101@HADOOP.LOCAL, \r\nServer not found in Kerberos database\r\n<\/pre>\n<p>When the reverse lookup record is present in DNS, you will see the following&#8230;<\/p>\n<pre>\r\nFeb 25 09:06:48 hdp.howard.local krb5kdc[1416](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 192.168.56.101: \r\nISSUE: authtime 1424871759, etypes {rep=18 tkt=18 ses=18}, zookeeper\/hdp.howard.local@HADOOP.LOCAL \r\nfor zookeeper\/hdp.howard.local@HADOOP.LOCAL\r\n<\/pre>\n<p>There was an <a href=\"https:\/\/issues.apache.org\/jira\/browse\/ZOOKEEPER-1666\" target=_blank>issue<\/a> for reverse lookups that was performance related, but I don&#8217;t think it is the same issue.<\/p>\n<p>Regardless, the org\/apache\/zookeeper\/ClientCnxn class snippet below is the one that does the call&#8230;<\/p>\n<pre>\r\nprivate void startConnect() throws IOException {\r\n  if(!isFirstConnect){\r\n    try {\r\n      Thread.sleep(r.nextInt(1000));\r\n    } catch (InterruptedException e) {\r\n        LOG.warn(\"Unexpected exception\", e);\r\n    }\r\n            }\r\n    state = States.CONNECTING;\r\n\r\n    InetSocketAddress addr;\r\n    if (rwServerAddress != null) {\r\n      addr = rwServerAddress;\r\n      rwServerAddress = null;\r\n    } else {\r\n        addr = hostProvider.next(1000);\r\n    }\r\n\r\n    setName(getName().replaceAll(\"\\\\(.*\\\\)\",\r\n              \"(\" + addr.getHostName() + \":\" + addr.getPort() + \")\"));\r\n    if (ZooKeeperSaslClient.isEnabled()) {\r\n      try {\r\n        String principalUserName = System.getProperty(\r\n               ZK_SASL_CLIENT_USERNAME, \"zookeeper\");\r\n        zooKeeperSaslClient = new ZooKeeperSaslClient(principalUserName+\"\/\"+addr.getHostName());\r\n      } catch (LoginException e) {\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Suffice it to say, on zookeeper 3.4.6 (at least) you will have all kinds of grief in a kerberos secured cluster if the zookeeper nodes can&#8217;t be looked up by passing the IP to get the hostname. Specifically, in the&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"more-link\" href=\"http:\/\/appcrawler.com\/wordpress\/2015\/02\/25\/zookeeper-and-reverse-lookups\/\">Read more &rarr;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[17],"tags":[],"_links":{"self":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/4725"}],"collection":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/comments?post=4725"}],"version-history":[{"count":8,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/4725\/revisions"}],"predecessor-version":[{"id":4733,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/4725\/revisions\/4733"}],"wp:attachment":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/media?parent=4725"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/categories?post=4725"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/tags?post=4725"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}