Hadoop Hdfs 配置 挂载hdfs文件系统（二）

Hadoop Hdfs 配置 挂载hdfs文件系统（二）

[align=left] [/align]

环境

[align=left] OS Centos 5.3[/align]
[align=left] Jdk1.6[/align]
[align=left] Hadoop-2.20.1[/align]
[align=left] Fuse-2.8.1[/align]
[align=left] Apache-ant-1.7.1-bin.tar.gz[/align]
[align=left] Ant是一种基于Java的build工具。理论上来说,它有些类似于(Unix)C中的make[/align]

1、Fuse安装

tar zxvf fuse-2.8.1.tar.gz
cd fuse2.8.1
./configure --prefix=/usr
make
make install
modprobe fuse
[align=left] [/align]

vi /etc/sysconfig/modules/my.modules
#!/bin/sh
modprobe fuse >/dev/null 2>&1
chmod +x /etc/sysconfig/modules/my.modules
[align=left] [/align]

2、Ant安装

wget http://www.meisei-u.ac.jp/mirror/apache/dist/ant/binaries/apache-ant-1.7.1-bin.tar.gz tar zxvf apache-ant-1.7.1-bin.tar.g
mv apache-ant-1.7.1-bin.tar.g /usr/local/ant
[align=left] [/align]

3、环境变量设置

export JAVA_HOME=/usr/local/1dk1.6
export HADOOP_HOME=/data/soft/hadoop-2.20.1
export OS_ARCH=i386
export OS_BIT=32 //如果系统是64位的，这里应该写64
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:${HADOOP_HOME}/build/c++/Linux-$OS_ARCH-$OS_BIT/lib:/usr/local/lib:/usr/lib
export ANT_HOME=/usr/local/ant
export PATH=$ANT_HOME/bin:$PATH
[align=left] [/align]

4、libhdfs 制作

cd $HADOOP_HOME/
ant compile-c++-libhdfs -Dlibhdfs=1 -Dcompile.c++=1
ln -s c++/Linux-$OS_ARCH-$OS_BIT/lib build/libhdfs
[align=left] [/align]

5、fuse-dfs

cd $HADOOP_HOME
ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
[align=left] [/align]

6、fuse配置

export JAVA_HOME=/usr/local/jdk1.6
export HADOOP_HOME=/data/soft/hadoop-2.20.1
export HADOOP_CONF_DIR=/data/soft/hadoop-0.20.1/conf
export OS_ARCH=i386
export OS_BIT=32
//以上都是新增加的内容

if [ "$HADOOP_HOME" = "" ]; then
export HADOOP_HOME=/usr/local/share/hadoop
fi
export PATH=$HADOOP_HOME/contrib/fuse_dfs:$PATH
for f in ls $HADOOP_HOME/lib/*.jar $HADOOP_HOME/*.jar ; do
export CLASSPATH=$CLASSPATH:$f
done
if [ "$OS_ARCH" = "" ]; then
export OS_ARCH=amd64
fi
if [ "$JAVA_HOME" = "" ]; then
export JAVA_HOME=/usr/local/java
fi
if [ "$LD_LIBRARY_PATH" = "" ]; then
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:/usr/local/share/hdfs/libhdfs/:/usr/local/lib
fi
./fuse_dfs $@ //修改成以下内容
fuse_dfs $@ //这里要把前面的./去掉,要不然挂载hdfs文件系统会找不到fuse_dfs文件
[align=left] [/align]

7、设置文件权限

chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh
chmod +x /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs
ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs_wrapper.sh /usr/local/bin
ln -s /data/soft/hadoop-2.20.1/build/contrib/fuse-dfs/fuse_dfs /usr/local/bin/
mkdir /mnt/dfs
[align=left] [/align]

8、手动挂载hdfs文件系统

fuse_dfs_wrapper.sh dfs://192.168.1.11:54310 /mnt/dfs

9、开机自动挂载hdfs文件系统

vi /etc/fstab
fuse_dfs_wrapper.sh dfs://192.168.1.11:54310 /mnt/dfs fuse rw,auto 0 0
[align=left]参考:http://www.pginjp.org/modules/newbb/viewtopic.php?topic_id=229&forum=22&post_id=2041#forumpost2041[/align]
[align=left] [/align]本文出自 “网海过客” 博客，请务必保留此出处http://cqfish.blog.51cto.com/622299/208404