Read note of HDFS User Guide
2014-03-26 21:16
344 查看
CheckPoint Node:
The Checkpoint node's memory requirements are on the same order as the NameNode. The Checkpoint node is started by (execute on checkpoint node)
two configuration parameters
dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not
been reached.
Backup Node:
As the Backup node maintains a copy of the namespace in memory,its RAM requirements are the same as the NameNode.
The NameNode supports one Backup node at a time.No
Checkpoint nodes may be registered if a Backup node is in use.
The Backup node is started by (execute on Backup node)
Import CheckPoint
The latest checkpoint can be imported to the NameNode if all other copies of the image and the edits files are lost. In order to do that one should:
Create an empty directory specified in the dfs.namenode.name.dir configuration variable;
Specify the location of the checkpoint directory in the configuration variable dfs.namenode.checkpoint.dir;
and start the NameNode with -importCheckpoint option.
Balancer all datanode
To start:
bin/hadoop-daemon.sh start balancer [-threshold <threshold>]
Example:
To stop:
Recovery mode
However, what can you do if the only storage locations available are corrupt? In this case, there is a special NameNode startup mode called Recovery mode that may allow you to recover
most of your data.
You can start the NameNode in recovery mode like so: namenode -recover
Recovery mode can cause you to lose data, you should always back up your edit log and fsimage before using it.
HDFS Upgrade and RollBack
Before upgrading, administrators need to remove existing backup using bin/hadoop dfsadmin -finalizeUpgrade command. The following briefly describes the typical upgrade procedure:
Before upgrading Hadoop software, finalize if there an existing backup. dfsadmin -upgradeProgress status can tell if the cluster needs to be finalized.
Stop the cluster and distribute new version of Hadoop.
Run the new version with -upgrade option (bin/start-dfs.sh -upgrade).
Most of the time, cluster works just fine. Once the new HDFS is considered working well (may be after a few days of operation), finalize the upgrade.Note that until
the cluster is finalized, deleting the files that existed before the upgrade does not free up real disk space on the DataNodes.
If there is a need to move back to the old version,
stop the cluster and distribute earlier version of Hadoop.
start the cluster with rollback option. (bin/start-dfs.h -rollback).
The Checkpoint node's memory requirements are on the same order as the NameNode. The Checkpoint node is started by (execute on checkpoint node)
bin/hdfs namenode -checkpoint
two configuration parameters
dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not
been reached.
Backup Node:
As the Backup node maintains a copy of the namespace in memory,its RAM requirements are the same as the NameNode.
The NameNode supports one Backup node at a time.No
Checkpoint nodes may be registered if a Backup node is in use.
The Backup node is started by (execute on Backup node)
bin/hdfs namenode -backup
Import CheckPoint
The latest checkpoint can be imported to the NameNode if all other copies of the image and the edits files are lost. In order to do that one should:
Create an empty directory specified in the dfs.namenode.name.dir configuration variable;
Specify the location of the checkpoint directory in the configuration variable dfs.namenode.checkpoint.dir;
and start the NameNode with -importCheckpoint option.
Balancer all datanode
To start:
bin/hadoop-daemon.sh start balancer [-threshold <threshold>]
Example:
bin/hadoop-daemon.sh start balancerstart the balancer with a default threshold of 10%
bin/hadoop-daemon.sh start balancer -threshold 5start the balancer with a threshold of 5%
To stop:
bin/hadoop-daemon.sh stop balancer
Recovery mode
However, what can you do if the only storage locations available are corrupt? In this case, there is a special NameNode startup mode called Recovery mode that may allow you to recover
most of your data.
You can start the NameNode in recovery mode like so: namenode -recover
Recovery mode can cause you to lose data, you should always back up your edit log and fsimage before using it.
HDFS Upgrade and RollBack
Before upgrading, administrators need to remove existing backup using bin/hadoop dfsadmin -finalizeUpgrade command. The following briefly describes the typical upgrade procedure:
Before upgrading Hadoop software, finalize if there an existing backup. dfsadmin -upgradeProgress status can tell if the cluster needs to be finalized.
Stop the cluster and distribute new version of Hadoop.
Run the new version with -upgrade option (bin/start-dfs.sh -upgrade).
Most of the time, cluster works just fine. Once the new HDFS is considered working well (may be after a few days of operation), finalize the upgrade.Note that until
the cluster is finalized, deleting the files that existed before the upgrade does not free up real disk space on the DataNodes.
If there is a need to move back to the old version,
stop the cluster and distribute earlier version of Hadoop.
start the cluster with rollback option. (bin/start-dfs.h -rollback).
相关文章推荐
- User's Guide Of Dagger2
- HDFS UserGuide
- Studying note of GCC-3.4.6 source (162 - continue)
- The problem of the user is not associated with a trusted sql server connection 混合登录选项设置的问题
- Vue报错:Uncaught TypeError: Cannot assign to read only property’exports‘ of object’#<Object>‘的解决方法
- READING NOTE: Pushing the Limits of Deep CNNs for Pedestrian Detection
- Studying note of GCC-3.4.6 source (21)
- Studying note of GCC-3.4.6 source (23)
- Studying note of GCC-3.4.6 source (173)
- some of the properties associated with the solution could not be read解决方法
- 【转】Gradle User Guide 中文版
- 【原创】The solutional manual of the Verilog HDL: A Guide to Digital Design and Synthesis (2nd)—ch07-II
- 【转】Gradle User Guide 中文版
- Vue components Cannot read property '__ob__' of undefined
- User-Note for human error
- 通过阅读Oracle Enterprise Asset Management User Guide,我对Oracle eAM的初步理解
- NVIDIA PhysX SDK 3.3.4 Documentation User's Guide翻译 --The PhysX API
- google-json2.0的UserGuide
- note of android
- sqoop 中文文档 User guide 二 import续