Elasticsearch Restore失灵

       最近，在备份Elasticsearch集群上发现其snapshot的restore功能失灵了，一直无法restore主集群的snapshot，每次都在调用创建repository API的时候失败。GET ~/_snapshot API返回如下的内容。而通过Azure
Explorer检查相应的Azure Storage Account，snapshot-0401-72d8早已经被清除了。

{

  "auto-restore-repo" : {

    "type" : "azure",

    "settings" : {

      "container" : "snapshot-0401-72d8",

      "location_mode" : "secondary_only",

      "compress" : "true",

      "account" : "elasticsearch"

    }

  }

}

进一步检查Elasticsearch的日志，发现了下面相关错误信息：

[2016-04-12 00:02:29,526][DEBUG][cluster.service        ] [ESNode-IN_0] processing [putsitory [auto-restore-repo]]: execute

[2016-04-12 00:02:29,526][WARN ][repositories             ] [ESNode-IN_0] failed to create repository [auto-restore-repo]

org.elasticsearch.ElasticsearchIllegalStateException: trying to modify or unregister repository that is currently used

 at org.elasticsearch.repositories.RepositoriesService.ensureRepositoryNotInUse(RepositoriesService.java:420)

 at org.elasticsearch.repositories.RepositoriesService.access$000(RepositoriesService.java:55)

 at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:108)

 at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)

 at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:204)

 at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:167)

 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

 at java.lang.Thread.run(Thread.java:745)

      
感觉是在什么地方卡住了，尝试删除auto-restore-repo并没有成功，重启了所有master节点也没有解决问题。最后，只好重启了整个集群才解决了这个问题。我们所用的版本是 1.7.2，在问题发生之前，Azure曾经对我们集群进行了日常update升级维护，很有可能就是这次维护的时候造成了restore操作卡住了，一直无法自动恢复。