Hadoop集群的回收机制

GVM(垃圾回收机制)

原理:当在集群中删除目录文件,集群自动在/user zkpk/.Trash/Current的集群目录中保存一定时间(根据ccore参数而定),在改时间内可以读对文件恢复,一旦超出规定时间,则无法恢复.

在Hadoop集群中有一下四个配置文件,他们都有自己的默认文件如下所示.

当启动集群或者重启后,会依次加载default.xml和site.xml如果二者有冲突以site.xml准

core-site.xml-->core-default.xml

yarn-site.xml-->yarn-default.xml

hdfs-site.xml--hdfs-default.xml

mapred-site.xml--mapred-default.xml

一 在 hadoop-2.5.2搜索xml配置文件

[zkpk@master ~]$ cd /home/zkpk/hadoop-2.5.2

[zkpk@master hadoop-2.5.2]$ find . |grep xml

显示(部分):

./share/doc/hadoop/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

./share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml

./share/doc/hadoop/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

三 进入core-default.xm

[zkpk@master hadoop-2.5.2]$ vim ./share/doc/hadoop/hadoop-project-dist/hadoop-common/core-default.xml

copy

<property>

<name>fs.trash.interval</name>

<value>1140</value>

<description>Number of minutes after which the checkpoint

gets deleted. If zero, the trash feature is disabled.

This option may be configured both on the server and the

client. If trash is disabled server side then the client

side configuration is checked. If trash is enabled on the

server side then the value configured on the server is

used and the client configuration value is ignored.

</description>

</property>

到core-site.xml中

四 测试:

1在集群中创建文件

[zkpk@master ~]$ hdfs fs -mkdir /input

2导入数据

3删除目录

[zkpk@master ~]$ hdfs fs -rmr /input

当在集群中创建文件或目录时,在集群中的user目录下生成 zkpk/.Trash/Current/input/sogou.10k

该文件保存时间为core-site.xml中是指的时间以分钟为单位

4 查看:

[zkpk@master ~]$ hadoop fs -lsr /user/zkpk

内容:

-rw-r--r-- 3 zkpk supergroup 1141839 2015-11-02 01:54 /user/zkpk/.Trash/Current/input/sogou.10k

五 恢复

[zkpk@master ~]$ hadoop fs -mv /user/zkpk/.Trash/Current/input /input

六 恢复查看

[zkpk@master ~]$ hadoop fs -ls /input

内容:

-rw-r--r-- 3 zkpk supergroup 1141839 2015-11-02 01:54 /input/sogou.10k

相关连接:http://www.cnblogs.com/ggjucheng/archive/2012/04/18/2454683.html