ES HDFS快照坑之could not read repository data from index blob

ES HDFS快照是遇到的坑

ERROR现场

当时快照的仓库已经建好了，但是创建snapshot时候报错,如下信息：

[my_hdfs_repo] could not read repository data from index blob

完整Logs

[2016-12-12T11:38:04,417][WARN ][r.suppressed             ] path: /_snapshot/my_hdfs_repo/20161209-snapshot, params: {repository=my_hdfs_repo, snapshot=20161209-snapshot}
org.elasticsearch.transport.RemoteTransportException: [node-2][10.90.6.234:9340][cluster:admin/snapshot/create]
Caused by: org.elasticsearch.repositories.RepositoryException: [my_hdfs_repo] could not read repository data from index blob
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:751) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.snapshots.SnapshotsService.createSnapshot(SnapshotsService.java:226) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.masterOperation(TransportCreateSnapshotAction.java:82) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.action.admin.cluster.snapshots.create.TransportCreateSnapshotAction.masterOperation(TransportCreateSnapshotAction.java:41) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:520) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.0.1.jar:5.0.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_92]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_92]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]

解决过程

于是乎各种Google，得出的结论是java的权限问题，需要修改Elasticsearch安装目录下的plugin-security.policy文件。如，我的安装目录是：/data2/elasticsearch-5.3.0/plugins/repository-hdfs/plugin-security.policy

修改文件

首先修改配置，添加java权限管理信息，配置后的配置信息如下

grant {
// Hadoop UserGroupInformation, HdfsConstants, PipelineAck clinit
permission java.lang.RuntimePermission "getClassLoader";

// UserGroupInformation (UGI) Metrics clinit
permission java.lang.RuntimePermission "accessDeclaredMembers";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

// org.apache.hadoop.util.StringUtils clinit
permission java.util.PropertyPermission "*", "read,write";

// org.apache.hadoop.util.ShutdownHookManager clinit
permission java.lang.RuntimePermission "shutdownHooks";

// JAAS is used always, we use a fake subject, hurts nobody
permission javax.security.auth.AuthPermission "getSubject";
permission javax.security.auth.AuthPermission "doAs";
permission javax.security.auth.AuthPermission "modifyPrivateCredentials";
## 注意，以下为我添加的信息
permission java.lang.RuntimePermission "accessDeclaredMembers";
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "shutdownHooks";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission javax.security.auth.AuthPermission "doAs";
permission javax.security.auth.AuthPermission "getSubject";
permission javax.security.auth.AuthPermission "modifyPrivateCredentials";
permission java.security.AllPermission;
permission java.util.PropertyPermission "*", "read,write";
permission javax.security.auth.PrivateCredentialPermission "org.apache.hadoop.security.Credentials * \"*\"", "read";
};

修改jvm配置

注意，这还没有结束，不知道怎么的，添加了plugin的配置，他没有生效。需要手动配置一次conf/jvm.options文件，在conf/jvm.options文件中添加以下信息：

-Djava.security.policy=file:////data2/elasticsearch-5.3.0/plugins/repository-hdfs/plugin-security.policy

验证

好了，到现在，你需要重启集群所有节点，然后try again。然后问题神奇的解决了。

curl -XPUT -u elastic 'http://192.168.81.2:9200/_snapshot/backup_hdfs/snapshot?wait_for_completion=true' -d '{
"indices": "*",
"ignore_unavailable": true,
"include_global_state": true
}'

{"snapshot":{"snapshot":"snapshot_1","uuid":"SprY4aHXTE6crhi5duJGAQ","version_id":5000199,"version":"5.0.1","indices":["ttst","test"],"state":"SUCCESS","start_time":"2017-03-16T07:23:54.568Z","start_time_in_millis":1489649034568,"end_time":"2017-03-16T07:24:03.961Z","end_time_in_millis":1489649043961,"duration_in_millis":9393,"failures":[],"shards":{"total":10,"failed":0,"successful":10}}}