Memory Issues in Hadoop

This topic describes the memory issues in Hadoop and YARN, which are as listed here:

Job History Server Memory Issues

While running Hadoop jobs, if you get the org.apache.hadoop.ipc.RemoteException, which is as mentioned below.

Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.OutOfMemoryError): GC overhead limit exceeded

at org.apache.hadoop.ipc.Client.call(Client.java:1471)
at org.apache.hadoop.ipc.Client.call(Client.java:1402)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy48.getJobReport(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)

Error Implication: The above exception implies that it has occurred due to the out-of-memory issue in the Job History Server (JHS).

Solution: It is recommended to use a larger coordinator node with at least 60G RAM where you can use 4GB heap memory. If you still face the issue, then you can increase the JHS memory by using the node bootstrap.

Increase the JHS’ memory by adding the following script in the node bootstrap.

#increase the JHS memory to 8G
sudo echo 'export HADOOP_JOB_HISTORYSERVER_HEAPSIZE="8192"' >> /etc/hadoop/mapred-env.sh

#restart the JHS
sudo -u mapred /usr/lib/hadoop2/sbin/mr-jobhistory-daemon.sh stop historyserver
sudo -u mapred /usr/lib/hadoop2/sbin/mr-jobhistory-daemon.sh start historyserver

YARN ApplicationMaster Exceeding its Physical Memory Limit Error

While running a Hadoop job, you may get this error on the ApplicationMaster’s physical memory limit.

Application application_<XXXXXXXXXXXXXX> failed 1 times due to AM Container for appattempt_<XXXXXXXXXXXXX> exited
with exitCode: -104

Diagnostics: Container pid=<XXXX>,containerID=container_<XXXXXXXXXXXXX> is running beyond physical memory limits.
Current usage: 5.2 GB of 4.8 GB physical memory used; 27.5 GB of 10.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_<XXXXXXXXXXXXX>

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.

Error implication:

The error implies that the YARN container is running beyond its physical memory limits.

Solution:

YARN provides the processing capacity to each application by allocating containers. A container is the basic unit of processing capacity in YARN and it is an encapsulation of resource elements (memory, cores, and so on). This issue may get resolved by configuring the appropriate value of the below parameters:

  • yarn.app.mapreduce.am.command-opts=-Xmx<XXXX>m. It sets the JVM arguments for an Application Master.

  • yarn.app.mapreduce.am.resource.mb=<XXXX>. It sets the container size.

Note

The container size must be greater than the heap size that is the value of the yarn.app.mapreduce.am.command-opts parameter.

Handling Exceeded Physical Memory Limit Error in a Mapper

When a mapper exceeds its physical memory limit, you would see this error in its logs. For information on the logs’ location, see Accessing the Logs of Hadoop Components.

Application application_<XXXXXXXXXXXXXX> failed 1 times due to AM Container for appattempt_<XXXXXXXXXXXXX> exited with exitCode: -104

Diagnostics: Container pid=<XXXX>,containerID=container_<XXXXXXXXXXXXX> is running beyond physical memory limits.
Current usage: 5.2 GB of 4.8 GB physical memory used; 27.5 GB of 10.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_<XXXXXXXXXXXXX>

Solution: Try increasing the mapper’s container and heap memory using these configuration properties.

mapreduce.map.java.opts = -Xmx<XXXX>m;
mapreduce.map.memory.mb = <XXXX>;

Handling Exceeded Physical Memory Limit Error in a Reducer

When a reducer exceeds its physical memory limit, you would see this error in its logs. For information on the logs’ location, see Accessing the Logs of Hadoop Components.

Application application_<XXXXXXXXXXXXXX> failed 1 times due to AM Container for appattempt_<XXXXXXXXXXXXX> exited with exitCode: -104

Diagnostics: Container pid=<XXXX>,containerID=container_<XXXXXXXXXXXXX> is running beyond physical memory limits.
Current usage: 5.2 GB of 4.8 GB physical memory used; 27.5 GB of 10.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_<XXXXXXXXXXXXX>

Solution: Try increasing the reducer’s container and heap memory using this configuration.

mapreduce.reduce.java.opts = -Xmx<XXXX>;
mapreduce.map.memory.mb = <XXXX>;

Handling the Java Heap Space Error

Exception: java.lang.OutOfMemoryError: Java heap space

If you are seeing this error in the mapper task, try to increase the mapper memory.

set mapreduce.map.memory.mb=<XXXX>;
set mapreduce.map.java.opts=-Xmx<XXXX>m;

If you are seeing this error in the reducer task, try to increase the reducer memory.

set mapreduce.reduce.memory.mb=<XXXX>;
set mapreduce.reduce.java.opts=-Xmx<XXXX>m;

Hadoop Client-side Memory Issues

If you get the Java out of the heap space or exceeded Garbage Collection (GC) overhead limit errors on the client node, while running the Hadoop job, then increase the client JVM heap size for the job by setting the HADOOP_CLIENT_OPTS variable as mentioned in this example.

sed -i "s/HADOOP_CLIENT_OPTS=\"-Xmx512m/HADOOP_CLIENT_OPTS=\"-Xmx4000m/g" /usr/lib/hadoop2/etc/hadoop/hadoop-env.sh

When you get this exception, Failed to sanitize XML document destined for handler class while running Hadoop jobs, you can imply that it has occurred due to the low client-side memory. You can increase the client-side memory as shown in this example:

export HADOOP_CLIENT_OPTS=-Xmx1024m