您的位置:首页 > 运维架构 > Apache

用Maven编译Apache flume-ng 1.5.0源码及问题解决

2014-08-25 19:17 387 查看
在今年的5月22号,Flume-ng1.5.0版本正式发布,关于Flume-ng1.5.0版本的新特性可以参见本博客的《Apache
Flume-ng1.5.0正式发布》进行了解。关于Apacheflume-ng1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng1.4.0及错误解决》。本文将讲述如何用Maven编译Apache
flume-ng1.5.0源码。

一、到官方网站下载相应版本的flume-ng源码


1
#
wgethttp:
//mirror.bit.edu.cn/apache/flume/1.5.0/apache-flume-1.5.0-src.tar.gz
2
#
tar-zxfapache-flume-
1.5
.
0
-src.tar.gz
3
#
cdapache-flume-
1.5
.
0
-src

二、用maven在相应的Hadoop上编译Flume-ng1.5.0源码


Flume-ng1.5.0默认的Hadoop版本是1.0.1,Hbase是0.92.1。我们可以在编译的时候加上一些参数,指定Hadoop2.2.0版本进行编译:

1
-Phadoop-
2

三、Hadoop、Hbase对应的test不能下载


  如果在编译的时候出现了类似如下的情况,请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉:

01
<!--
02
User:
过往记忆
03
Date:
14
-
6
-
16
04
Time:
23
:
45
05
bolg:
http:
//www.iteblog.com
06
本文地址:http:
//www.iteblog.com/archives/1043
07
过往记忆博客,专注于hadoop、hive、spark、shark、flume的技术博客,大量的干货
08
过往记忆博客微信公共帐号:iteblog_hadoop
09
-->
10
<dependency>
11
<groupId>org.apache.hbase</groupId>
12
<artifactId>hbase</artifactId>
13
<version>${hbase.version}</version>
14
<classifier>tests</classifier>
15
<scope>test</scope>
16
</dependency>
17
18
<dependency>
19
<groupId>org.apache.hadoop</groupId>
20
<artifactId>hadoop-test</artifactId>
21
<version>${hadoop.version}</version>
22
</dependency>

四、elasticsearch-0.90.1.jarread错误


  如果你在编译的过程中出现一下异常:

1
[ERROR]
Failedtoexecutegoalorg.apache.maven.plugins:
2
maven-compiler-plugin:
2.3
.
2
:compile
(
default
-compile)
on
3
project
flume-ng-elasticsearch-sink:Compilationfailure
4
[ERROR]
error:errorreading
5
/org/elasticsearch/elasticsearch/
0.90
.
1
/elasticsearch-
0.90
.
1
.jar;
6
error
inopeningzipfile
7
[ERROR]
->[Help
1
]
  只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除,然后重新下载即可:

1
#
rmelasticsearch-
0.90
.
1
.jar
2
wget
http:
//central.maven.org/maven2/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar

五、ua-parser-1.3.0下载失败


编译过程中,下载ua-parser-1.3.0.pom可能失败,出现如下错误:

01
[ERROR]
Failedtoexecutegoalonprojectflume-ng-morphline-solr-sink:
02
Could
notresolvedependencies
for

project
03
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:
1.5
.
0
:
04
Failed
tocollectdependencies
for

[org.apache.flume:flume-ng-core:jar:
1.5
.
0
05
(compile),
org.slf4j:slf4j-api:jar:
1.6
.
1

(compile),
06
org.kitesdk:kite-morphlines-all:pom:
0.12
.
0

(compile?),
07
org.slf4j:jcl-over-slf4j:jar:
1.6
.
1

(provided),
08
org.apache.solr:solr-test-framework:jar:
4.3
.
0

(test),
09
org.kitesdk:kite-morphlines-solr-core:jar:tests:
0.12
.
0

(test),
10
junit:junit:jar:
4.10


(test)]:Failedtoreadartifactdescriptor
11
for

ua_parser:ua-parser:jar:
1.3
.
0
:
Couldnottransferartifact
12
ua_parser:ua-parser:pom:
1.3
.
0


from/tomaven-twttr(http:
//maven.twttr.com):
13
Connection
tohttp:
//maven.twttr.com
refused:Connectiontimedout->[Help1]
只需要加一个新的repository即可:

<repository>

<id>p2.jfrog.org</id>

<url>http://p2.jfrog.org/libs-releases</url>

</repository>

假如上面的repository仍然不能用,可以google“ua-parser-1.3.0.pom”获取其他源。


六、编译Flume-ng1.5.0源码


1
#
mvninstall-Phadoop-
2


-DskipTests-Dtar
编译需要一段时间,这个需要看你编译时的网速。如果一切顺利的话,你将会看到以下输出,说明你的Flume-ng1.5.0编译成功:

01
[INFO]
------------------------------------------------------------------------
02
[INFO]
ReactorSummary:
03
[INFO]
04
[INFO]
ApacheFlume......................................SUCCESS[
1
.360s]
05
[INFO]
FlumeNGSDK......................................SUCCESS[
2
.903s]
06
[INFO]
FlumeNGConfiguration............................SUCCESS[
0
.483s]
07
[INFO]
FlumeNGCore.....................................SUCCESS[
2
.531s]
08
[INFO]
FlumeNGSinks....................................SUCCESS[
0
.110s]
09
[INFO]
FlumeNGHDFSSink................................SUCCESS[
1
.668s]
10
[INFO]
FlumeNGIRCSink.................................SUCCESS[
0
.484s]
11
[INFO]
FlumeNGChannels.................................SUCCESS[
0
.074s]
12
[INFO]
FlumeNGJDBCchannel.............................SUCCESS[
0
.579s]
13
[INFO]
FlumeNGfile-basedchannel.......................SUCCESS[
0
.712s]
14
[INFO]
FlumeNGSpillableMemorychannel.................SUCCESS[
0
.552s]
15
[INFO]
FlumeNGNode.....................................SUCCESS[
0
.619s]
16
[INFO]
FlumeNGEmbeddedAgent...........................SUCCESS[
1
.001s]
17
[INFO]
FlumeNGHBaseSink...............................SUCCESS[
2
.373s]
18
[INFO]
FlumeNGElasticSearchSink.......................SUCCESS[
0
.899s]
19
[INFO]
FlumeNGMorphlineSolrSink......................SUCCESS[
2
.483s]
20
[INFO]
FlumeNGKiteDatasetSink........................SUCCESS[
35
.925s]
21
[INFO]
FlumeSources.....................................SUCCESS[
0
.054s]
22
[INFO]
FlumeScribeSource...............................SUCCESS[
1
.231s]
23
[INFO]
FlumeJMSSource..................................SUCCESS[
2
.133s]
24
[INFO]
FlumeTwitterSource..............................SUCCESS[
1
.204s]
25
[INFO]
FlumelegacySources..............................SUCCESS[
0
.043s]
26
[INFO]
FlumelegacyAvrosource..........................SUCCESS[
1
.234s]
27
[INFO]
FlumelegacyThriftSource........................SUCCESS[
1
.320s]
28
[INFO]
FlumeNGClients..................................SUCCESS[
0
.038s]
29
[INFO]
FlumeNGLog4jAppender...........................SUCCESS[
6
.175s]
30
[INFO]
FlumeNGTools....................................SUCCESS[
1
.559s]
31
[INFO]
FlumeNGdistribution.............................SUCCESS[
1
:
51
.993s]
32
[INFO]
FlumeNGIntegrationTests........................SUCCESS[
1
.517s]
33
[INFO]
------------------------------------------------------------------------
34
[INFO]
BUILDSUCCESS
35
[INFO]
------------------------------------------------------------------------
36
[INFO]
Totaltime:
3
:
03
.921s
37
[INFO]
Finishedat:MonJun
16

23
:
37
:
51


CST
2014
38
[INFO]
FinalMemory:135M/745M
39
[INFO]
------------------------------------------------------------------------
最后将会在flume-ng-dist/target目录下产生以下的文件:

查看源代码

打印帮助

1
total
29208
2
drwxr-xr-x
3


wypwyp
4096


Jun
16

23
:
36

apache-flume-
1.5
.
0
-bin
3
-rw-r--r--
1


wypwyp
27307850


Jun
16

23
:
36

apache-flume-
1.5
.
0
-bin.tar.gz
4
drwxr-xr-x
3


wypwyp
4096


Jun
16

23
:
36

apache-flume-
1.5
.
0
-src
5
-rw-r--r--
1


wypwyp
2543174


Jun
16

23
:
37

apache-flume-
1.5
.
0
-src.tar.gz
6
drwxr-xr-x
2


wypwyp
4096


Jun
16

23
:
36

archive-tmp
7
drwxr-xr-x
3


wypwyp
4096


Jun
16

23
:
36

maven-shared-archive-resources
8
-rw-r--r--
1


wypwyp
990


Jun
16

23
:
37

rat.txt
apache-flume-1.5.0-bin这个文件夹里面的东西可以直接拿来用。
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签: