hadoop错误-Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream
2016-03-04 09:00
686 查看
设置set mapreduce.job.reduces=68;把reduce调大,会出错?
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#3 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) Container killed by
the ApplicationMaster. Container killed on request. Exit code is 143
reduce步骤内存不足?
解决方法(度娘):调低mapreduce.reduce.shuffle.memory.limit.percent的值
默认为0.25 现在调成0.10?
没调整这个参数,把大SQL变成多个小的SQL就没出现这个情况,待观察
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#3 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165) Container killed by
the ApplicationMaster. Container killed on request. Exit code is 143
reduce步骤内存不足?
解决方法(度娘):调低mapreduce.reduce.shuffle.memory.limit.percent的值
默认为0.25 现在调成0.10?
没调整这个参数,把大SQL变成多个小的SQL就没出现这个情况,待观察
相关文章推荐
- apache 花生壳 搭建个人网站 搭建实录
- centos下添加环境变量和启动apache
- JSONArray().fromObject(); 出现org.apache.catalina.core.StandardWrapperValve invoke错误的解决办法
- 安装配置PHP Apachec 错误处理
- Apache shiro 笔记整理之整合spring
- LAMP第二部分apache配置
- 全文检索-Apache Lucene初探
- apache编译出错 error: mod_deflate has been requested
- Unable to load config class org.apache.struts2.interceptor.debugging.DebuggingInterceptor at interce
- class "org.apache.commons.collections.FastHashMap"'s signer information does not match signer inform
- Apache 的ab测试
- AXIS客户端调用出错org.apache.axis.NoEndPointException
- Apache shiro 笔记整理之加密
- Apache Kafka:下一代分布式消息系统
- (1) PHP 随笔---跟老韩学(24-34)---Apache、PHP简介、环境的搭建
- 在本地使用Apache配置WEB服务器
- Eclipse使用Maven创建Web时错误:Could not resolve archetype org.apache.maven.archetypes:maven-archetype-webap
- org.apache.subversion.javahl.ClientException: Previous operation has not finished
- Apache Thrift - 可伸缩的跨语言服务开发框架
- 闰年2月29号 通过apache的ftp工具从ftp上下载文件失败