Common Errors/Exceptions

[ProTechSkills]

Issue: NodeManager Service of two of my datanodes (slave1 and slave2) was stopping after some time of running start-all.sh command at namenode (master), whereas the NodeManager service of datanode running at “master” was working fine.

> I cross checked the logs(/opt/hadoop-2.2.0/logs/yarn-hadoop-resourcemanager-slave1.log) of respective nodemanagers and found Connection Refused Exception.

> I checked the yarn-site.xml files of both datanodes and changed the entries of resourcemanager to “master” instead of slave1 or slave2 as resourcemanager was running on namenode (master).
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>

Finally it worked. 🙂

Error/Exception : Connection Refused
(Reference : http://wiki.apache.org/hadoop/ConnectionRefused)

You get a ConnectionRefused Exception when there is a machine at the address specified, but there is no program listening on the specific TCP port the client is using -and there is no firewall in the way silently dropping TCP connection requests. If you do not know what a TCP connection request is, please consult the specification.

Unless there is a configuration error at either end, a common cause for this is the Hadoop service isn’t running.

This stack trace is very common when the cluster is being shut down -because at that point Hadoop services are being torn down across the cluster, which is visible to those services and applications which haven’t been shut down themselves. Seeing this error message during cluster shutdown is not anything to worry about.

If the application or cluster is not working, and this message appears in the log, then it is more serious.

Check the hostname the client using is correct
Check the IP address the client is trying to talk to for the hostname is correct.
Make sure the destination address in the exception isn’t 0.0.0.0 -this means that you haven’t actually configured the client with the real address for that
service, and instead it is picking up the server-side property telling it to listen on every port for connections.

Check that there isn’t an entry for your hostname mapped to 127.0.0.1 or 127.0.1.1 in /etc/hosts (Ubuntu is notorious for this)
Check the port the client is trying to talk to using matches that the server is offering a service on.
On the server, try a telnet localhost <port> to see if the port is open there.
On the client, try a telnet <server> <port> to see if the port is accessible remotely.
Try connecting to the server/port from a different machine, to see if it just the single client misbehaving.
If you are using a Hadoop-based product from a third party, including those from Cloudera, Hortonworks, Intel, EMC and others -please use the support channels provided by the vendor.
Please do not file bug reports related to your problem, as they will be closed as Invalid
None of these are Hadoop problems, they are host, network and firewall configuration issues. As it is your cluster, only you can find out and track down the problem.

• This topic was modified 9 years, 3 months ago by ProTechSkills.

[Author]

Posts