elk,logstash ->elastcisearch 429

logstash log出现

01:44:50.412 [[main]>worker14] INFO  logstash.outputs.elasticsearch - retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@45d0f0be on EsThreadPoolExecutor[bulk, queue capacity = 100, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6bf39a64[Running, pool size = 32, active threads = 32, queued tasks = 60, completed tasks = 7071143]]"})

说明

es 线程队列已满,不能再处理新请求了.

处理

我先修改了es的jvm参数 -Xss1m -> Xss10m

elasticsearch.yml修改bulk.queue大小 thread_pool.bulk.queue_size: 100 .从默认60修改为100.

未能解决.logstash继续出log,这次的 queued tasks = 100.

再查询资料.可能是logstash发送速度过快导致.

修改logstash.yml配置

默认

#   pipeline:
#     batch:
#       size: 125
#       delay: 5
#

改为

pipeline.batch.size: 1500
pipeline.batch.delay: 200

重启logstash,没有这样日志出现了

参考资料: https://blog.csdn.net/u013613428/article/details/77963604