Hadoop “Unable to load native-hadoop library for y
2015-10-12 14:36
211 查看
http://devsolvd.com/questions/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos
The answer depends... I just installed Hadoop 2.6 from tarball on 64-bit CentOS 6.6. The Hadoop install did indeed come with a prebuilt 64-bit native library. For my install, it is here:
And I know it is 64-bit:
Unfortunately, I stupidly overlooked the answer right there staring me in the face as I was focuses on, "Is this library 32 pr 64 bit?":
So, lesson learned. Anyway, the rest at least led me to being able to suppress the warning. So I continued and did everything recommended in the other answers to provide the library path using the HADOOP_OPTS environment variable to no avail. So I looked at the source code. The module that generates the error tells you the hint (util.NativeCodeLoader):
So, off to here to see what it does:
http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.action/0.2.6/org/apache/hadoop/util/NativeCodeLoader.java/
Ah, there is some debug level logging - let's turn that on a see if we get some additional help. This is done by adding the following line to $HADOOP_CONF_DIR/log4j.properties file:
Then I ran a command that generates the original warning, like stop-dfs.sh, and got this goodie:
And the answer is revealed in this snippet of the debug message (the same thing that the previous ldd command 'tried' to tell me:
What version of GLIBC do I have? Here's simple trick to find out:
So, can't update my OS to 2.14. Only solution is to build the native libraries from sources on my OS or suppress the warning and just ignore it for now. I opted to just suppress the annoying warning for now (but do plan to build from sources in the future) buy using the same logging options we used to get the debug message, except now, just make it ERROR level.
I hope this helps others see that a big benefit of open source software is that you can figure this stuff out if you take some simple logical steps.
The answer depends... I just installed Hadoop 2.6 from tarball on 64-bit CentOS 6.6. The Hadoop install did indeed come with a prebuilt 64-bit native library. For my install, it is here:
/opt/hadoop/lib/native/libhadoop.so.1.0.0
And I know it is 64-bit:
[hadoop@VMWHADTEST01 native]$ ldd libhadoop.so.1.0.0 ./libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./libhadoop.so.1.0.0)linux-vdso.so.1 => (0x00007fff43510000)libdl.so.2 => /lib64/libdl.so.2 (0x00007f9be553a000)libc.so.6 => /lib64/libc.so.6 (0x00007f9be51a5000)/lib64/ld-linux-x86-64.so.2 (0x00007f9be5966000)
Unfortunately, I stupidly overlooked the answer right there staring me in the face as I was focuses on, "Is this library 32 pr 64 bit?":
`GLIBC_2.14' not found (required by ./libhadoop.so.1.0.0)
So, lesson learned. Anyway, the rest at least led me to being able to suppress the warning. So I continued and did everything recommended in the other answers to provide the library path using the HADOOP_OPTS environment variable to no avail. So I looked at the source code. The module that generates the error tells you the hint (util.NativeCodeLoader):
15/06/18 18:59:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
So, off to here to see what it does:
http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.action/0.2.6/org/apache/hadoop/util/NativeCodeLoader.java/
Ah, there is some debug level logging - let's turn that on a see if we get some additional help. This is done by adding the following line to $HADOOP_CONF_DIR/log4j.properties file:
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=DEBUG
Then I ran a command that generates the original warning, like stop-dfs.sh, and got this goodie:
15/06/18 19:05:19 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: /opt/hadoop/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/hadoop/lib/native/libhadoop.so.1.0.0)
And the answer is revealed in this snippet of the debug message (the same thing that the previous ldd command 'tried' to tell me:
`GLIBC_2.14' not found (required by opt/hadoop/lib/native/libhadoop.so.1.0.0)
What version of GLIBC do I have? Here's simple trick to find out:
[hadoop@VMWHADTEST01 hadoop]$ ldd --versionldd (GNU libc) 2.12
So, can't update my OS to 2.14. Only solution is to build the native libraries from sources on my OS or suppress the warning and just ignore it for now. I opted to just suppress the annoying warning for now (but do plan to build from sources in the future) buy using the same logging options we used to get the debug message, except now, just make it ERROR level.
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
I hope this helps others see that a big benefit of open source software is that you can figure this stuff out if you take some simple logical steps.
相关文章推荐
- flume 采集数据到hdfs
- Sqoop--关系型数据库跟hdfs数据传输工具
- HDFS简介
- Hadoop学习笔记-随手记
- 【转】HDFS 安全模式
- HDFS文件的合并
- HDFD 四个配置文件(core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml )的简单介绍
- flume+kafka+hdfs详解
- 【转】HDFS 的Trash回收站功能的配置、使用
- HDFS原理及其个元素功能
- 认识HDFS分布式文件系统
- 搭建hdfsHA时遇到的问题及其解决方案
- HDFS-Architecture剖析
- flume学习(五):flume将log4j日志数据写入到hdfs
- HDFS序列化
- 解决从本地文件系统上传到HDFS时的权限问题
- 解决从本地文件系统上传到HDFS时的权限问题
- HDFS基本命令
- hadoop2.7.0升级到2.7.1,版本升级
- 解读HDFS