(2) flume 入门学习 HelloWorld 及HDFS 遇到的问题 总结
2014-12-15 16:31
666 查看
(1) HelloWorld
An agent is started using a shell script called flume-ng which is located in the bin directory of the Flume distribution. You need to specify the agent name, the config directory, and the config
file on the command line:
Now the agent will start running source and sinks configured in the given properties file.
Here, we give an example configuration file, describing a single-node Flume deployment. This configuration lets a user generate events and subsequently logs them to the console.
This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console.
The configuration file names the various components, then describes their types and configuration parameters. A given configuration file might define several named agents; when a given Flume process is launched a flag is passed telling it which named agent
to manifest.
Given this configuration file, we can start Flume as follows:
Note that in a full deployment we would typically include one more option: --conf=<conf-dir>.
The <conf-dir> directory would include a shell script flume-env.sh and potentially a log4j properties file. In
this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
From a separate terminal, we can then telnet port 44444 and send Flume an event:
The original Flume terminal will output the event in a log message.
Congratulations - you’ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(2)单节点 Flume 直接写入 HDFS
查阅文章:http://my.oschina.net/leejun2005/blog/288136#OSC_h2_10 按照3.2章节练习(flume版本1.5.2)
遇到问题:
▂ ▃ ▄ ▅ ▆ ▇ █
2014-12-15 17:22:28,418 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)]
Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:251)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 12 more
意思就是:在类路径下找不到什么什么calss.....应该是缺少jar包了 。网上折腾了半天,官网的说明也看了。
单节点flume写入HDFS中 用到的sink是 HDFS Sink。
This sink writes events into the Hadoop
Distributed File System (HDFS). It currently supports creating text and sequence files. It supports compression in both file types. The files can be rolled (close current file and create a new one) periodically based on the elapsed time or size of data or
number of events. It also buckets/partitions data by attributes like timestamp or machine where the event originated. The HDFS directory path may contain formatting escape sequences that will replaced by the HDFS sink to generate a directory/file name to store
the events.
Using this sink requires hadoop to be installed so that Flume can use the Hadoop jars to communicate with the HDFS cluster. Note that a version of Hadoop that supports the sync() call is required.
大意就是:使用这种sink需要hadoop的jar包。由于也没说具体哪个jar包
(1)org/apache/hadoop/io/SequenceFile$CompressionType 是在hadoop-2.3.0-cdh5.1.0\share\hadoop\common\hadoop-common-2.3.0-cdh5.1.0.jar中 , 将该jar包放到flume的lib包中。再起动还是报同样的错。
(2)查看env (服务器装了两个flume A配置了FLUME_HOME,B我练习的没有配置),于是进到conf中修改flume-env.sh
J***A_HOME=/home/xxx/jdk/jdk1.7.0_55(原来已有)
FLUME_HOME=/home/xxx/pjmflume/flume(此处添加)
(3)再起报错 Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
缺少了 commons-configuration-1.6.jar 包。添上。
(4)再起报错 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
缺少 hadoop-auth-2.3.0-cdh5.1.0.jar 包。添上。
(5)报错
2014-12-15 17:42:34,356 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)]
HDFS IO error
java.io.IOException: No FileSystem for scheme: hdfs
缺少 hadoop-hdfs-2.3.0-cdh5.1.0.jar 包。添上
----------------------------------------------------------------------------
理论上 就是flume的 hdfs sink需要用到hadoop的上述jar包。
懒得一个一个错误的排查可直接将hadoop share下的所有jar包 拷贝一份移至
flume的lib中即可。
----------------------------------------------------------------------------
▂ ▃ ▄ ▅ ▆
▇ █
包不少了 又报错
org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:428)] HDFSWriter is already
closed
java.net.NoRouteToHostException: No Route to Host from gz-asp-nginx-10802106/127.0.0.1 to 10.111.15.4:9111 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost
Caused by: java.net.NoRouteToHostException: No route
to host
保通信错误: 猜测应该是 本机 与 hdfs集群不在一个网络上。
telnet ip及端口信息
telnet 10.211.151.4 9111
Trying 10.211.151.4...
telnet: connect to address 10.211.151.4: No route to host
telnet: Unable to connect to remote host: No route to host
最后换了台可以hdfs交互的服务器练习。可以成功。
-----------------------------------------------------------------------------------------------------------------------
Starting an agent
An agent is started using a shell script called flume-ng which is located in the bin directory of the Flume distribution. You need to specify the agent name, the config directory, and the configfile on the command line:
$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
Now the agent will start running source and sinks configured in the given properties file.
A simple example
Here, we give an example configuration file, describing a single-node Flume deployment. This configuration lets a user generate events and subsequently logs them to the console.# example.conf: A single-node Flume configuration # Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console.
The configuration file names the various components, then describes their types and configuration parameters. A given configuration file might define several named agents; when a given Flume process is launched a flag is passed telling it which named agent
to manifest.
Given this configuration file, we can start Flume as follows:
$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Note that in a full deployment we would typically include one more option: --conf=<conf-dir>.
The <conf-dir> directory would include a shell script flume-env.sh and potentially a log4j properties file. In
this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
From a separate terminal, we can then telnet port 44444 and send Flume an event:
$ telnet localhost 44444 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. Hello world! <ENTER> OK
The original Flume terminal will output the event in a log message.
12/06/19 15:32:19 INFO source.NetcatSource: Source starting 12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] 12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D Hello world!. }
Congratulations - you’ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(2)单节点 Flume 直接写入 HDFS
遇到问题:
▂ ▃ ▄ ▅ ▆ ▇ █
2014-12-15 17:22:28,418 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)]
Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:251)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 12 more
意思就是:在类路径下找不到什么什么calss.....应该是缺少jar包了 。网上折腾了半天,官网的说明也看了。
单节点flume写入HDFS中 用到的sink是 HDFS Sink。
This sink writes events into the Hadoop
Distributed File System (HDFS). It currently supports creating text and sequence files. It supports compression in both file types. The files can be rolled (close current file and create a new one) periodically based on the elapsed time or size of data or
number of events. It also buckets/partitions data by attributes like timestamp or machine where the event originated. The HDFS directory path may contain formatting escape sequences that will replaced by the HDFS sink to generate a directory/file name to store
the events.
Using this sink requires hadoop to be installed so that Flume can use the Hadoop jars to communicate with the HDFS cluster. Note that a version of Hadoop that supports the sync() call is required.
大意就是:使用这种sink需要hadoop的jar包。由于也没说具体哪个jar包
(1)org/apache/hadoop/io/SequenceFile$CompressionType 是在hadoop-2.3.0-cdh5.1.0\share\hadoop\common\hadoop-common-2.3.0-cdh5.1.0.jar中 , 将该jar包放到flume的lib包中。再起动还是报同样的错。
(2)查看env (服务器装了两个flume A配置了FLUME_HOME,B我练习的没有配置),于是进到conf中修改flume-env.sh
J***A_HOME=/home/xxx/jdk/jdk1.7.0_55(原来已有)
FLUME_HOME=/home/xxx/pjmflume/flume(此处添加)
(3)再起报错 Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
缺少了 commons-configuration-1.6.jar 包。添上。
(4)再起报错 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
缺少 hadoop-auth-2.3.0-cdh5.1.0.jar 包。添上。
(5)报错
2014-12-15 17:42:34,356 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)]
HDFS IO error
java.io.IOException: No FileSystem for scheme: hdfs
缺少 hadoop-hdfs-2.3.0-cdh5.1.0.jar 包。添上
----------------------------------------------------------------------------
理论上 就是flume的 hdfs sink需要用到hadoop的上述jar包。
懒得一个一个错误的排查可直接将hadoop share下的所有jar包 拷贝一份移至
flume的lib中即可。
----------------------------------------------------------------------------
▂ ▃ ▄ ▅ ▆
▇ █
包不少了 又报错
org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:428)] HDFSWriter is already
closed
java.net.NoRouteToHostException: No Route to Host from gz-asp-nginx-10802106/127.0.0.1 to 10.111.15.4:9111 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost
Caused by: java.net.NoRouteToHostException: No route
to host
保通信错误: 猜测应该是 本机 与 hdfs集群不在一个网络上。
telnet ip及端口信息
telnet 10.211.151.4 9111
Trying 10.211.151.4...
telnet: connect to address 10.211.151.4: No route to host
telnet: Unable to connect to remote host: No route to host
最后换了台可以hdfs交互的服务器练习。可以成功。
-----------------------------------------------------------------------------------------------------------------------
相关文章推荐
- [项目过程中所遇到的各种问题记录]学习篇——对工作以来的学习过的开源项目进行总结—动软代码生成工具
- 在自己学习stuts是遇到问题后总结的经验。
- cocos2d x 入门学习(二)Star Fighter项目交叉编译时遇到的问题
- 学习JMF框架遇到的问题及总结
- Flume-ng-1.4.0安装及运行遇到问题总结
- 在学习ASP.net3.5从入门到精通中遇到问题1
- djang框架学习中遇到的问题总结
- 使用Hadoop API操作HDFS文件时遇到的问题及解决总结
- cocos2d-X入门(win7+VS2012环境配置以及学习中遇到的问题)
- .net学习遇到的问题总结
- 在学习ASP.net3.5从入门到精通中遇到问题2
- 学习ubuntu遇到的问题总结,陆续更新。。。
- MFC下OpenGL编程入门遇到的几个问题总结[1]
- 在学习ASP.net3.5从入门到精通中遇到问题1
- 在自己学习stuts是遇到问题后总结的经验。
- WTL学习过程中遇到的一些问题总结
- (转)WTL学习过程中遇到的一些问题总结
- mahout分类学习和遇到的问题总结
- C++ 关于static variables的学习中遇到的问题总结