logstash輸出帶有序列號的彈性搜索索引

我在 Logstash 7.10 中使用 AWS Elastic Search（版本 7.10）。目的是將內容從 logstash 發送到彈性搜索，并使用策略在特定大小或時間后滾動索引。

policy: {
    "policy_id": "Rollover_Policy",
    "description": "roller index",
    "last_updated_time": 1634910129219,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "1mb"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm"
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "replica_count": {
                        "number_of_replicas": 1
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "1h"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "products*"
            ],
            "priority": 100,
            "last_updated_time": 1634910129219
        }
    ]
}

當我嘗試在 logstash 輸出插件中將 ilm_enabled 設定為 true 時，它??無法與彈性搜索 xpack API 連接。

注意：AWS 彈性搜索不支持 xpack 和 ILM。

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => true
        ilm_rollover_alias => "products"
        ilm_pattern => "{now/d}-000001"
        ilm_policy => "Rollover_Policy"
}

所以我已將 ilm_enabled 標志更改為 false 并嘗試了以下選項

elasticsearch {
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false
        index => "products-%{ YYYY.MM.dd}-000001"
}

現在的問題是，即使在翻轉之后，logstash 仍然將檔案發送到 001 索引而不是新索引。如果我不在索引名稱中給出 -000001，那么翻轉就會失敗。

uj5u.com熱心網友回復：

使用以下 REST 請求在彈性中創建索引。由于索引名稱具有日期模式，翻轉將創建具有當前日期的新索引。

PUT 
{
  "settings":{
    "number_of_shards":1,
    "number_of_replicas":1
  },
  "aliases": {
    "products":  {
      "is_write_index": true
    }
  }

為索引模式創建模板以及翻轉別名

PUT _index_template/products_logs
{
  "index_patterns": [
    "products*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "opendistro": {
        "index_state_management": {
          "rollover_alias": "products"
        }
      }
    }
  }
}

在logstash輸出插件中提供以下詳細資訊以將資料發送到彈性搜索

elasticsearch {  
        hosts => "${elasticsearch_endpoint}"
        user => "${elasticsearch_user}"
        password => "${elasticsearch_password}"
        ilm_enabled => false 
        index => "products"
}

注意：索引名代表索引的別名。

標籤：亚马逊网络服务 弹性搜索 日志存储

上一篇：Elasticsearch5.6.16補丁log4j與2.16

下一篇：Elasticsearch停止令牌過濾器不起作用