Native Libraries Guide
2016-01-14 09:04
387 查看
Native Hadoop Library
Hadoop has native implementations of certain components for performance reasons and for non-availability of Java implementations. These components are available in a single, dynamically-linked native library called the native hadoop library. On the *nix platforms the library is named libhadoop.so.Usage
It is fairly easy to use the native hadoop library:Review the components.
Review the supported platforms.
Either download a hadoop release, which will include a pre-built version of the native hadoop library, or build your own version of the native hadoop library. Whether you download or build, the name for the library is the same: libhadoop.so
Install the compression codec development packages (>zlib-1.2, >gzip-1.2):
If you download the library, install one or more development packages - whichever compression codecs you want to use with your deployment.
If you build the library, it is mandatory to install both development packages.
Check the runtime log files.
Components
The native hadoop library includes various components:Compression Codecs (bzip2, lz4, snappy, zlib)
Native IO utilities for HDFS Short-Circuit Local Reads and Centralized Cache Management in HDFS
CRC32 checksum implementation
Supported Platforms
The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:
RHEL4/Fedora
Ubuntu
Gentoo
On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.
Build
The native hadoop library is written in ANSI C and is built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). This means it should be straight-forward to build the library on any platform with a standards-compliant C compiler and the GNU autotools-chain (see the supported platforms).The packages you need to install on the target platform are:
C compiler (e.g. GNU C Compiler)
GNU Autools Chain: autoconf, automake, libtool
zlib-development package (stable version >= 1.2.0)
openssl-development package(e.g. libssl-dev)
Once you installed the prerequisite packages use the standard hadoop pom.xml file and pass along the native flag to build the native hadoop library:
$ mvn package -Pdist,native -DskipTests -Dtar
You should see the newly-built library in:
$ hadoop-dist/target/hadoop-2.7.1/lib/native
Please note the following:
It is mandatory to install both the zlib and gzip development packages on the target platform in order to build the native hadoop library; however, for deployment it is sufficient to install just one package if you wish to use only one codec.
It is necessary to have the correct 32/64 libraries for zlib, depending on the 32/64 bit jvm for the target platform, in order to build and deploy the native hadoop library.
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html
相关文章推荐
- iOS中的UINavigationController导航栏视图控制器的简单应用
- Android UI布局之用户界面概述
- UITextView的使用详解
- 破解 ue 30天试用方法
- easyui全选、取消所有页
- iOS开发-UITableView顶部图片下拉放大
- Android UI基础之Dialog对话框的使用
- Android UI之AutoCompleteTextView
- AndroidUI基础之Spinner的使用
- Divide and conquer:Aggressive Cows(POJ 2456)
- UIDynamic—UIKit 动力学
- 【UE4学习】03——Blueprint快速入门
- Jfinal 2.1 集成 Guice,实现注解功能, 加入Service接口和实现类,直接晒代码
- UIView常用属性的注意点:
- 【iOS开发】值对象--NSValue,NSNumber
- 解决相片上传主键唯一性问题,java 上传相片到服务器的时候,相片保存主键用guid。
- request response
- UIImagePickerController详解
- 关于NGUI动态加载图片,并且实现动态打图集的过程
- serialVersionUID的作用