Hive2.1安装后运行mapreduce出错，换成hadoop2.6.4全程记录！

Hive2.1安装后运行mapreduce出错，换成hadoop2.6.4全程记录！

解压hadoop2.6.4到/opt目录后。

修改hadoop-env.sh

# The java implementation to use.

export JAVA_HOME=/usr/java/jdk

修改~/.bashrc
#HADOOP VARIABLES START
export JAVA_HOME=/usr/java/jdk
export HADOOP_INSTALL=/opt/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export HIVE_HOME=/opt/hive
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib/native"
export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
#HADOOP VARIABLES END
export PATH=$PATH:$JAVA_HOME/bin:$HIVE_HOME/bin:$HADOOP_INSTALL/bin:$SPARK_HOME/bin

 如果把HADOOP_OPTS配置为：

HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib" 
 
 则会出现如下错误：       

[html] view plain copy 在CODE上查看代码片派生到我的代码片

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  

在hadoop根目录下执行Jar命令，运行自带的wordcount

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount input output
结果出来了：
16/11/23 12:46:36 INFO mapreduce.Job: Counters: 33
File System Counters
FILE: Number of bytes read=547468
FILE: Number of bytes written=1054406
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework

说明Hadoop单机模式配置成功！下面，进入伪分布式模式配置。

在hadoop根目录下，新建tmp、hdfs、hdfs/data、hdfs/name目录。
脚本如下：

dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir tmp
dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir hdfs
dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir hdfs/data
dyq@ubuntu:/opt/hadoop-2.6.4$ mkdir hdfs/name
dyq@ubuntu:/opt/hadoop-2.6.4$ ls
bin hdfs input libexec NOTICE.txt README.txt share
etc include lib LICENSE.txt output sbin tmp
core-site.xml配置如下：
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.0.10:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-2.6.4/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
</configuration>

hdfs-site.xml配置：这里的/opt/hadoop-2.6.4/hdfs/name是我们刚才新建的几个目录。

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop-2.6.4/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop-2.6.4/hdfs/data</value>
</property>
<property>    //这个属性节点是为了防止后面eclopse存在拒绝读写设置的
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>

yarn-env.sh

# some Java parameters
export JAVA_HOME=/usr/java/jdk
mapred-env.sh

# some Java parameters
export JAVA_HOME=/usr/java/jdk

mapreduce-site.xml:

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!--指定JobTracker主机与端口号-->
<property>
<name>mapred.job.tracker</name>
<value>ubuntu:9001</value>
</property>
</configuration>

修改yarn-site.xml：

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>ubuntu</value>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

格式化namenode：

hadoop namenode -format

返回成功：

16/11/23 13:29:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-302189630-192.168.0.10-1479878959376
16/11/23 13:29:19 INFO common.Storage: Storage directory /opt/hadoop/hdfs/name has been successfully formatted.
16/11/23 13:29:19 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/11/23 13:29:19 INFO util.ExitUtil: Exiting with status 0
16/11/23 13:29:19 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/192.168.0.10
************************************************************/

启动hadoop：
sbin/start-dfs.sh
dyq@ubuntu:/opt/hadoop$ sbin/start-dfs.sh
16/11/23 13:30:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [ubuntu]
ubuntu: starting namenode, logging to /opt/hadoop-2.6.4/logs/hadoop-dyq-namenode-ubuntu.out
localhost: starting datanode, logging to /opt/hadoop-2.6.4/logs/hadoop-dyq-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.6.4/logs/hadoop-dyq-secondarynamenode-ubuntu.out
16/11/23 13:30:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
dyq@ubuntu:/opt/hadoop$ jps
3016 SecondaryNameNode
3123 Jps
2826 DataNode
2721 NameNode


启动yarn试试：

dyq@ubuntu:/opt/hadoop$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.6.4/logs/yarn-dyq-resourcemanager-ubuntu.out
localhost: starting nodemanager, logging to /opt/hadoop-2.6.4/logs/yarn-dyq-nodemanager-ubuntu.out
dyq@ubuntu:/opt/hadoop$ jps
3167 ResourceManager
3016 SecondaryNameNode
3497 Jps
2826 DataNode
2721 NameNode
3274 NodeManager
dyq@ubuntu:/opt/hadoop$

hadoop web控制台页面的端口整理：

50070：hdfs文件管理

8088：ResourceManager

8042：NodeManager

运行wordcount程序：

建立目录：hadoop dfs -mkdir  /input

hadoop dfs -put README.txt /input/

在hadoop主目录中运行：

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.4.jar wordcount /input/README.txt /output/1124

如果出现错误：

1/1 local-dirs are bad: /opt/hadoop/tmp/nm-local-dir; 1/1 log-dirs are bad: /opt/hadoop/logs/userlogs 

则是临时文件夹设置有误，重新设置正确即可！