hadoop命令行运行wordcount程序 web端监控不到作业状态

在命令行或者是Eclipse远程连接运行Wordcount程序后，在页面8088端口号监控，始终没有变化，显示application为空。

说明作业只是成功连接到了hdfs，但是并没有用到集群中mapreduce服务。

修改集群配置文件yarn-site.xml，加上如下几句：

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

重启运行wordcount，页面有作业显示，如下



但是作业运行失败，报错：

14/11/06 16:31:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/11/06 16:31:37 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:9001

14/11/06 16:31:38 INFO input.FileInputFormat: Total input paths to process : 23

14/11/06 16:31:38 INFO mapreduce.JobSubmitter: number of splits:23

14/11/06 16:31:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415262420184_0002

14/11/06 16:31:39 INFO impl.YarnClientImpl: Submitted application application_1415262420184_0002

14/11/06 16:31:39 INFO mapreduce.Job: The url to track the job: http://hadoop-05:8088/proxy/application_1415262420184_0002/
14/11/06 16:31:39 INFO mapreduce.Job: Running job: job_1415262420184_0002

14/11/06 16:31:43 INFO mapreduce.Job: Job job_1415262420184_0002 running in uber mode : false

14/11/06 16:31:43 INFO mapreduce.Job: map 0% reduce 0%

14/11/06 16:31:43 INFO mapreduce.Job: Job job_1415262420184_0002 failed with state FAILED due to: Application application_1415262420184_0002 failed 2 times due to AM Container for appattempt_1415262420184_0002_000002 exited with exitCode: -1 due to: No space
available in any of the local directories.

.Failing this attempt.. Failing the application.

14/11/06 16:31:43 INFO mapreduce.Job: Counters: 0

mapreduce.framework.name应该配置在mapred-site.xml中，修改配置文件：

在mapred-site.xml中添加：

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

在yarn-site.xml中将刚才的这一段配置删除，添加：

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

运行程序，web界面有显示：



但是程序卡住了

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.3.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.

It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

14/11/06 16:45:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/11/06 16:45:37 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:9001

14/11/06 16:45:38 INFO input.FileInputFormat: Total input paths to process : 23

14/11/06 16:45:38 INFO mapreduce.JobSubmitter: number of splits:23

14/11/06 16:45:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415263490341_0001

14/11/06 16:45:39 INFO impl.YarnClientImpl: Submitted application application_1415263490341_0001

14/11/06 16:45:39 INFO mapreduce.Job: The url to track the job: http://hadoop-05:8088/proxy/application_1415263490341_0001/
14/11/06 16:45:39 INFO mapreduce.Job: Running job: job_1415263490341_0001

这个时候，我已经想死了。

把yarn-site.xml中刚配置的那段删除。运行

报错：

14/11/06 18:13:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/11/06 18:13:05 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:9001

14/11/06 18:13:06 INFO input.FileInputFormat: Total input paths to process : 2

14/11/06 18:13:06 INFO mapreduce.JobSubmitter: number of splits:2

14/11/06 18:13:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415268754099_0001

14/11/06 18:13:07 INFO impl.YarnClientImpl: Submitted application application_1415268754099_0001

14/11/06 18:13:07 INFO mapreduce.Job: The url to track the job: http://hadoop-05:8088/proxy/application_1415268754099_0001/
14/11/06 18:13:07 INFO mapreduce.Job: Running job: job_1415268754099_0001

14/11/06 18:13:12 INFO mapreduce.Job: Job job_1415268754099_0001 running in uber mode : false

14/11/06 18:13:12 INFO mapreduce.Job: map 0% reduce 0%

14/11/06 18:13:12 INFO mapreduce.Job: Job job_1415268754099_0001 failed with state FAILED due to: Application application_1415268754099_0001 failed 2 times due to AM Container for appattempt_1415268754099_0001_000002 exited with exitCode: -1 due to: No space
available in any of the local directories.

.Failing this attempt.. Failing the application.

14/11/06 18:13:12 INFO mapreduce.Job: Counters: 0

网上没有找到任何解决方法！！！！！！！！！！

df -lh 查看了下centos环境的磁盘占用率，发现根目录的盘的占用率还真是100%，简直是邪门。

于是利用命令du -h dir挨个找，

找到大文件，将其删除，占用率降下来了。

继续运行程序，出现错误：

14/11/06 20:17:12 INFO mapreduce.Job: Task Id : attempt_1415268754099_0002_m_000001_1, Status : FAILED

Container launch failed for container_1415268754099_0002_01_000005 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152)

at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)

at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

看错误应该是没有在yarn-site.xml里面配置附属服务为mapreduce，所以还是要把之前那段加上。

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

终于大功告成啦！！！

结论：

mapreduce V2和V1不同，需要另外配置mapreduce的运行时环境，否则就默认在本地运行。

mapred-site.xml:

mapreduce.framwork.name 代表mapreduce的运行时环境，默认是local，需配置成yarn

yarn-site.xml:

yarn.nodemanager.aux-services代表附属服务的名称，如果使用mapreduce，则需要将其配置成mapreduce_shuffle（hadoop2.2.0之前的则配为mapreduce.shuffle）