关于hadoop 1.2.1版本不支持“dfs.support.append”参数的解决办法
2014-12-25 11:02
585 查看
最近在测试hadoop+fluentd方案,但是fluentd日志收集系统,需要append功能选择,写日志到HDFS中,官方给出的解决方案是:
修改hdfs-site.xml 文件,增加如下行:
但是format namenode节点的时候会报错:
[hadoop@node1 ~]$ hadoop namenode -format
14/12/25 10:35:25 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = node1.test.com/172.16.41.151
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_67
************************************************************/
14/12/25 10:35:25 INFO util.GSet: Computing capacity for map BlocksMap
14/12/25 10:35:25 INFO util.GSet: VM type = 64-bit
14/12/25 10:35:25 INFO util.GSet: 2.0% max memory = 932184064
14/12/25 10:35:25 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/12/25 10:35:25 INFO util.GSet: recommended=2097152, actual=2097152
14/12/25 10:35:26 INFO namenode.FSNamesystem: fsOwner=hadoop
14/12/25 10:35:26 INFO namenode.FSNamesystem: supergroup=supergroup
14/12/25 10:35:26 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/12/25 10:35:26 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/12/25 10:35:26 WARN namenode.FSNamesystem: The dfs.support.append option is in your configuration, however append is not supported. This configuration option is no longer required to enable sync
14/12/25 10:35:26 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/12/25 10:35:26 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/12/25 10:35:26 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/12/25 10:35:26 INFO common.Storage: Image file /usr/local/hadoop_tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/12/25 10:35:26 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:35:26 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:35:26 INFO common.Storage: Storage directory /usr/local/hadoop_tmp/dfs/name has been successfully formatted.
网上搜索半天,最好解决办法如下:
用一下语句:
然后再次格式化namenode,就不报错了!
[hadoop@node1 conf]$ hadoop namenode -format
14/12/25 10:47:18 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = node1.test.com/172.16.41.151
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_67
************************************************************/
Re-format filesystem in /usr/local/hadoop_tmp/dfs/name ? (Y or N) Y
14/12/25 10:47:21 INFO util.GSet: Computing capacity for map BlocksMap
14/12/25 10:47:21 INFO util.GSet: VM type = 64-bit
14/12/25 10:47:21 INFO util.GSet: 2.0% max memory = 932184064
14/12/25 10:47:21 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/12/25 10:47:21 INFO util.GSet: recommended=2097152, actual=2097152
14/12/25 10:47:21 INFO namenode.FSNamesystem: fsOwner=hadoop
14/12/25 10:47:21 INFO namenode.FSNamesystem: supergroup=supergroup
14/12/25 10:47:21 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/12/25 10:47:21 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/12/25 10:47:21 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/12/25 10:47:21 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/12/25 10:47:21 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/12/25 10:47:21 INFO common.Storage: Image file /usr/local/hadoop_tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/12/25 10:47:21 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:47:21 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:47:21 INFO common.Storage: Storage directory /usr/local/hadoop_tmp/dfs/name has been successfully formatted.
14/12/25 10:47:21 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1.test.com/172.16.41.151
************************************************************/
本文出自 “shine_forever的博客” 博客,请务必保留此出处http://shineforever.blog.51cto.com/1429204/1595776
修改hdfs-site.xml 文件,增加如下行:
<property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> <property> <name>dfs.support.append</name> <value>true</value> </property> <property> <name>dfs.support.broken.append</name> <value>true</value> </property>
但是format namenode节点的时候会报错:
[hadoop@node1 ~]$ hadoop namenode -format
14/12/25 10:35:25 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = node1.test.com/172.16.41.151
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_67
************************************************************/
14/12/25 10:35:25 INFO util.GSet: Computing capacity for map BlocksMap
14/12/25 10:35:25 INFO util.GSet: VM type = 64-bit
14/12/25 10:35:25 INFO util.GSet: 2.0% max memory = 932184064
14/12/25 10:35:25 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/12/25 10:35:25 INFO util.GSet: recommended=2097152, actual=2097152
14/12/25 10:35:26 INFO namenode.FSNamesystem: fsOwner=hadoop
14/12/25 10:35:26 INFO namenode.FSNamesystem: supergroup=supergroup
14/12/25 10:35:26 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/12/25 10:35:26 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/12/25 10:35:26 WARN namenode.FSNamesystem: The dfs.support.append option is in your configuration, however append is not supported. This configuration option is no longer required to enable sync
14/12/25 10:35:26 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/12/25 10:35:26 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/12/25 10:35:26 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/12/25 10:35:26 INFO common.Storage: Image file /usr/local/hadoop_tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/12/25 10:35:26 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:35:26 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:35:26 INFO common.Storage: Storage directory /usr/local/hadoop_tmp/dfs/name has been successfully formatted.
网上搜索半天,最好解决办法如下:
用一下语句:
<property> <name>dfs.support.broken.append</name> <value>true</value> </property>
然后再次格式化namenode,就不报错了!
[hadoop@node1 conf]$ hadoop namenode -format
14/12/25 10:47:18 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = node1.test.com/172.16.41.151
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.7.0_67
************************************************************/
Re-format filesystem in /usr/local/hadoop_tmp/dfs/name ? (Y or N) Y
14/12/25 10:47:21 INFO util.GSet: Computing capacity for map BlocksMap
14/12/25 10:47:21 INFO util.GSet: VM type = 64-bit
14/12/25 10:47:21 INFO util.GSet: 2.0% max memory = 932184064
14/12/25 10:47:21 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/12/25 10:47:21 INFO util.GSet: recommended=2097152, actual=2097152
14/12/25 10:47:21 INFO namenode.FSNamesystem: fsOwner=hadoop
14/12/25 10:47:21 INFO namenode.FSNamesystem: supergroup=supergroup
14/12/25 10:47:21 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/12/25 10:47:21 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/12/25 10:47:21 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/12/25 10:47:21 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/12/25 10:47:21 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/12/25 10:47:21 INFO common.Storage: Image file /usr/local/hadoop_tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/12/25 10:47:21 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:47:21 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop_tmp/dfs/name/current/edits
14/12/25 10:47:21 INFO common.Storage: Storage directory /usr/local/hadoop_tmp/dfs/name has been successfully formatted.
14/12/25 10:47:21 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1.test.com/172.16.41.151
************************************************************/
本文出自 “shine_forever的博客” 博客,请务必保留此出处http://shineforever.blog.51cto.com/1429204/1595776
相关文章推荐
- 关于“打开项目**时发生问题,尝试退出并重新启动应用程序。如果问题仍然存在,则可能是由于正在使用不支持的项目版本,或者项目文件可能损坏”的问题的解决办法。
- 关于unity5.2发布ios版本不支持ipv6网络-解决办法,升级版本
- 关于IE8 及以下版本不支持Map() 对象的解决办法
- 关于”因为此版本的应用程序不支持其项目类型(.csproj)“错误的解决办法
- 关于IE8版本提示“不支持‘trim’此属性或者方法”的解决办法。转摘雨网络
- Thin的DateChooser代码学习(关于js的函数参数为一个完整的函数以及“对象不支持此属性或方法”错误的解决)(原创,转载请声明)
- mysql4.1以上版本连接时出现Client does not support authentication protocol问题解决办法
- 关于 cuteEditor 6.2 版本 过期 的 解决 办法
- 关于ASP.NET回发或回调参数无效问题的解决办法
- c#中复制类提示“Resource”参数不支持重复项的解决办法
- 关于Monorail里Url中文参数的解决办法
- 关于用reader获取不到输出参数的解决办法
- 【转】关于Oracle 9i或9i以上版本客户端连接Oracle 8i及8i版本以下服务器端中文字符乱码的解决办法
- DEVEXPRESS 各个版本 关于加载不信任组件的解决办法
- 当客户端浏览器不支持相应版本的apple时自动下载运行环境JVM的解决办法!
- Oracle新手笔记(2) 关于Oracle 9i或9i以上版本客户端连接Oracle 8i及8i版本以下服务器端中文字符乱码的解决办法
- 当客户端浏览器不支持相应版本的apple时自动下载运行环境jvm的解决办法!
- mysql4.1以上版本连接时出现Client does not support authentication protocol #1251错误问题解决办法
- mysql4.1以上版本连接时出现Client does not support authentication protocol问题解决办法
- 安装Exchange 2007时误选"是否支持outlook 2003以前版本时"的解决办法