filebeat把nginx的json日志格式文件推送到es

#### 前言

为了方便在es中查看到nginx的日志，需要把nginx日志输送到es中，这里采用filebeat推送到es中

#### 准备工作

##### 部署及配置nginx的日志json格式

这里不再赘述，详见:https://blog.csdn.net/weixin_42715225/article/details/105603410

##### 部署filebeat

这里亦不再赘述，详见:https://blog.csdn.net/weixin_42715225/article/details/105601286   

注释: 其中filebeat.yml文件改为如下内容

```

cat > /opt/filebeat/filebeat.yml <<-EOF
filebeat.inputs:
- type: log
  enabled: true    # 必须为true，否则logstash接收不到
  paths:
    - /var/log/nginx/access.log    # 日志文件路径

  fields:
    index: "nginx-access_log-%{+yyyy.MM}"
  encoding: plain
  json.keys_under_root: true
  json.overwrite_keys: true
  json.message_key: log

close_inactive: 2m
scan_frequency: 2m
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["url/ip:9200"]    # 建议: 云主机上部署，url采用内网的url，同理，ip也是采用内网的ip

  indices:
    - index: "nginx-access_log-%{+yyyy.MM}"
      when.contains:
        fields:
          index: "nginx-access_log-%{+yyyy.MM}"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# 如下可参考/忽略

setup.template.name: "filebeat"
setup.template.path: "/mnt/filebeat/filebeat.template.json"
setup.template.pattern: "filebeat-*"
template.overwrite: True

EOF

```

补充:

···

cat > /mnt/filebeat/filebeat.template.json <<-EOF
{
  "mappings": {
    "_default_": {
      "_all": {
        "norms": false
      },
      "dynamic_templates": [
        {
          "fields": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string",
            "path_match": "fields.*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "beat": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "input_type": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "message": {
          "norms": false,
          "type": "text"
        },
        "offset": {
          "type": "long"
        },
        "source": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "tags": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "type": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    }
  },
  "order": 0,
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "filebeat-*"
}

EOF

···

• 点赞
• 收藏
• 分享
• 文章举报