yarn is running beyond physical memory limits 问题解决

yarn is running beyond physical memory limits 问题解决

Diagnostic Messages for this Task:

Container [pid=7830,containerID=container_1397098636321_27548_01_000297] is running beyond physical memory limits. Current usage: 2.1 GB of 2 GB physical memory used; 2.7
GB of 4.2 GB virtual memory used. Killing container.

Dump of the process-tree for container_1397098636321_27548_01_000297 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

|- 7830 7816 7830 7830 (java) 2547 390 2924818432 539150 /export/servers/jdk1.6.0_25/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2224m -Djava.io.tmpdir=/data2/nm/local/usercache/admin/appcache/application_1397098636321_27548/container_1397098636321_27548_01_000297/tmp
-Dlog4j.configuration=container-log4j.properties......

检查yarn-site-xml job内存限制
<property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<value>2048</value>

</property>

解决方法：
1.增加yarn.scheduler.minimum-allocation-mb内存上限。
2.--hiveconf mapred.child.java.opts=-Xmx？？？？m 一定要小于yarn.scheduler.minimum-allocation-mb

如果是vm超了，如下：调整yarn.nodemanager.vmem-pmem-ratio

查看log没有明显的ERROR，但存在类似以下描述的日志

2012-05-16 13:08:20,876 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 18, cluster_timestamp: 1337134318909, }, attemptId: 1, }, id: 6, }, state:
C_COMPLETE, diagnostics: "Container [pid=15641,containerID=container_1337134318909_0018_01_000006] is running beyond virtual memory limits. Current usage: 32.1mb of 1.0gb physical memory used; 6.2gb
of 2.1gb virtual memory used. Killing container.\nDump of the process-tree for container_1337134318909_0018_01_000006
:\n\t|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE\n\t|- 15641 26354 15641 15641 (java) 36 2 6686339072 8207 /home/zhouchen.zm/jdk1.6.0_23/bin/java

原因：

该错误是YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。调节比例值可以解决该问题。具体参数为：yarn-site.xml中的yarn.nodemanager.vmem-pmem-ratio

原文出自：/article/11633798.html