hive on tez Caused by: java.lang.OutOfMemoryError: Java heap space

昨天进行两个hive表关联导出数据，不幸的是爆出如下错误：

Status: Failed

Vertex failed, vertexName=Map 5, vertexId=vertex_1443634917922_0008_1_05, diagnostics=[Task failed, taskId=task_1443634917922_0008_1_05_000006, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space

    at

org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)

    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)

    at

org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)

    at

org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)

    at

org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:415)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

    at

org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)

    at

org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)

    at java.util.concurrent.FutureTask.run(FutureTask.java:262)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.OutOfMemoryError: Java heap space

仔细观看日志发现上面错误信息，主要是因为内存不足，有意思的是我整个服务器都在给它跑，怎么会内存不足呢？后来查阅资料和咨询我一个同事吴哥发现，这个内存不足是值java堆内存，好把，既然内存不足，那我就看看hive给的默认内存是多少

hive>SET
hive.tez.container.size;

hive.tez.container.size=6144;

hive>SET
hive.tez.java.opts;

hive.tez.java.opts=-server
-Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps

尴尬了，确实有点小，那么容器我给它20G，java.opt给80%容器试试，反正服务器内存大


SET hive.tez.container.size=20480;

SET hive.tez.java.opts=-Xmx16384m;

解决了

，真是天天踩坑。