Flume启动时报错Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.解决办法(图文详解)
2017-07-29 10:51
1291 查看
[b] 前期博客[/b]
这句话,说的很明显。
你的hdfs.calltimeout可能设置得太低或HDFS需要花费太长的时间。
我的是
这个大家,可以还改大些。
hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
为何这两个设置这么大。超时出现问题正常的设置小一点,根据情况自己调试下。
建议初始值:
hdfs.sinks.fs.hdfs.rollInterval=10
hdfs.sinks.fs.hdfs.rollSize=0
依次:
20
40
80
200
等值。
Flume自定义拦截器(Interceptors)或自带拦截器时的一些经验技巧总结(图文详解)
问题详情2017-07-29 10:30:12,784 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: SOURCE, name: fileSource. src.events.accepted == 21158 2017-07-29 10:30:12,786 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: SOURCE, name: fileSource. src.events.received == 21158 2017-07-29 10:30:12,786 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:177)] Shutdown Metric for type: SOURCE, name: fileSource. src.open-connection.count == 0 2017-07-29 10:30:12,959 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long. at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(BucketWriter.java:649) at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:409) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) 2017-07-29 10:30:13,152 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long. at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long. at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(BucketWriter.java:649) at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:409) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430) ... 3 more
这句话,说的很明显。
你的hdfs.calltimeout可能设置得太低或HDFS需要花费太长的时间。
我的是
#---------hdfsSink 相关配置------------------ agent1.sinks.hdfsSink.type = hdfs # 注意, 我们输出到下面一个子文件夹datax中 agent1.sinks.hdfsSink.hdfs.path = hdfs://master:9000/data/types/%Y%m%d/%{log_type} agent1.sinks.hdfsSink.hdfs.writeFormat = Text agent1.sinks.hdfsSink.hdfs.fileType = DataStream agent1.sinks.hdfsSink.hdfs.callTimeout = 3600000 agent1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true #当文件大小为52428800字节时,将临时文件滚动成一个目标文件 agent1.sinks.hdfsSink.hdfs.rollSize = 52428800 #events数据达到该数量的时候,将临时文件滚动成目标文件 agent1.sinks.hdfsSink.hdfs.rollCount = 0 #每隔N s将临时文件滚动成一个目标文件 agent1.sinks.hdfsSink.hdfs.rollInterval = 1200 #配置前缀和后缀 agent1.sinks.hdfsSink.hdfs.filePrefix=run
这个大家,可以还改大些。
agent1.sinks.hdfsSink.hdfs.callTimeout = 7200000
hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
为何这两个设置这么大。超时出现问题正常的设置小一点,根据情况自己调试下。
建议初始值:
hdfs.sinks.fs.hdfs.rollInterval=10
hdfs.sinks.fs.hdfs.rollSize=0
依次:
20
40
80
200
等值。
相关文章推荐
- java.lang.UnsupportedOperationException: setXIncludeAware is not supported on this JAXP implementation or earlier: class gnu.xml.dom.JAXPFactory的解决办法(图文详解)
- java.net.SocketTimeoutException: Read timed out解决办法
- tomcat servlet java.net.SocketTimeoutException: Read timed out 解决办法
- Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/common/base/Function : Unsupported major.minor version 52.0的解决办法(图文详解)
- 执行Hive时出现org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.NumberFormatException: For input string: "1s"错误的解决办法(图文详解)
- IDEA里运行代码时出现Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger的解决办法(图文详解)
- 基于Web的Kafka管理器工具之Kafka-manager启动时出现Exception in thread "main" java.lang.UnsupportedClassVersionError错误解决办法(图文详解)
- java.net.SocketTimeoutException: Read timed out 的解决办法 android平台 腾讯微博开发
- java.net.SocketTimeoutException: Read timed out 异常解决办法
- java.net.SocketTimeoutException: Read timed out 的解决办法
- ambari-server启动出现Caused by: java.lang.RuntimeException:java.lang.ClassNotFoundEception:com.mysql.jdbc.Driver问题解决办法(图文详解)
- java.lang.IllegalArgumentException: pointerIndex out of range两种解决办法
- java.lang.RuntimeException: ImageLoader must be init with configuration before using的解决方法
- Eclipse 报 “Exception in thread "main" java.lang.OutOfMemoryError: Java heap space ”错误的解决办法以及文本文件换行
- java.lang.IllegalArgumentException:pointerIndex out of range问题的两种解决办法
- java.lang.IllegalArgumentException: pointerIndex out of range 问题的两种解决办法
- Eclipse 报 “Exception in thread "main" java.lang.OutOfMemoryError: Java heap space ”错误的解决办法
- Eclipse 报 “Exception in thread "main" java.lang.OutOfMemoryError: Java heap space ”错误的解决办法
- 关于使用动态代理创建代理对象是报错 java.lang.ClassCastException: $Proxy0 cannot be cast to 的解决办法
- java.lang.ClassCastException: android.widget.ImageView cannot be cast to android.widget.TextView解决办法