集群搭建7.4.1版本,配置

3臺機器組成一個集群，分別為：a,b,c
a:
編輯a的config/elasticsearch.yml組態檔,修改后如下

# ======================== Elasticsearch Configuration #=========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster #—————————————————
#
# Use a descriptive name for your cluster:
#集群名稱
cluster.name: my-application
#
# ------------------------------------ Node ##
#
# Use a descriptive name for the node:
#確定master
node.master: true
#節點名稱
node.name: node-1
#
#discovery.zen.minimum_master_nodes: 3
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths #
#
# Path to directory where to store the data (separate multiple locations by comma):
#es資料存放位置，需要手動創建目錄和賦予權限
path.data: /opt/soft/data
#
# Path to log files:
#
#es日志存放位置，需要手動創建目錄和賦予權限
path.logs: /opt/soft/log
#
# ----------------------------------- Memory #
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network #
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#允許自身各種ip訪問
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#對外服務埠
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery #
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#指定集群里的所有節點，9300是集群間相互通信的埠
discovery.seed_hosts:  ["10.209.5.87:9300","10.209.5.88:9300","10.209.5.89:9300"]
#discovery.zen.ping.unicast.hosts: ["10.209.5.79","10.209.5.80","10.209.5.78"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#集群啟動指定的可選舉的master節點
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway #
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various #—————————————————
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#這兩行允許跨域
http.cors.enabled: true
http.cors.allow-origin: "*"
#reindex同步資料，資料遷移需要的其他機器的白名單，不然不能使用reindex,這表示當前節點可以#從以下白名單節點獲取資料，通常是其他集群的節點
reindex.remote.whitelist: ["10.209.5.84:9200","10.209.5.78:9200","10.209.1.48:9200","10.209.1.35:5200","10.47.187.45:5200","10.47.195.38:5200"]
#指定冷歸檔資料的存放位置目錄，冷歸檔的資料可以壓縮檔案夾后剪切移到其他機器，目錄需要手#動創建并賦予權限
path.repo: ["/opt/soft/es_backups/backups", "/opt/soft/es_backups/longterm_backups"]

b:
機器的elasticsearch.yml
其他一樣，修改
#注釋
#node.master: true
#節點名稱
node.name: node-2

c:
機器的elasticsearch.yml
其他一樣，修改
#注釋
#node.master: true
#節點名稱
node.name: node-3

修改每一臺機器的記憶體大小引數（64g為例）
修改config/jvm.options檔案,最大不能超過31g,最好不超過整個機器的記憶體50%
-Xms30g
-Xmx30g

可安裝ik分詞器
需要指定版本

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.4.1/elasticsearch-analysis-ik-7.4.1.zip

linux優化

關閉交換磁區，防止記憶體置換降低性能

swapoff -a

vim /etc/security/limits.conf

#文尾添加
* soft nofile 65535
* hard nofile 131072
* soft nproc 4096
* hard nproc 4096

vim /etc/sysctl.conf

vm.max_map_count=262145 

#重繪配置

sysctl -p 	

es不允許root啟動
#增加用戶

useradd esuser　

#切換用戶

su esuser 

啟動命令：
一定要檢查防火墻是否開放9200，9300埠
在解壓目錄執行命令

./bin/elasticsearch -d

索引映射創建，優化

創建索引es_persist_3

創建索引 es_persist_3
url

put http://ip:port/es_persist_3

json

{
  "settings": {
    "number_of_shards": "12",
    "number_of_replicas": "1",
    "index.translog.durability": "async",
    "index.translog.sync_interval": "60s",
    "index.translog.flush_threshold_size": "1024mb"
  }
}

創建映射mapping es_persist_3

創建mapping es_persist_3
url

post http://ip:port/es_persist_3/_mapping

json

{
  "properties": {
    "servCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "httpMethod": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "type": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "servVersionProxyType": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "exceptionStack": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "exceptionTime": {
      "type": "date"
    },
    "@version": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "host": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "pAppName": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "id": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "receiveSize": {
      "type": "long"
    },
    "authType": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "externalTime": {
      "type": "long"
    },
    "cAppName": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "returnSize": {
      "type": "long"
    },
    "authCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "statusDesc": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "platformTime": {
      "type": "long"
    },
    "servName": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "componentPort": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "esbId": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "responseSize": {
      "type": "long"
    },
    "message": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "logTime": {
      "type": "date"
    },
    "tags": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "receiveTime": {
      "type": "long"
    },
    "@timestamp": {
      "type": "date"
    },
    "messageList": {
      "properties": {
        "sizeX": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "serialNumber": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "header": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "time": {
          "type": "long"
        },
        "body": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "type": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "url": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      }
    },
    "componentHost": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "cAppCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "fromIp": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "complete": {
      "type": "boolean"
    },
    "requestSize": {
      "type": "long"
    },
    "logtime": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "callTime": {
      "type": "long"
    },
    "pAppCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "statusCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    }
  }
}

創建索引 es_persist_4

url

put http://ip:port/es_persist_34

json

{
  "settings": {
    "number_of_shards": "2",
    "number_of_replicas": "1",
    "index.translog.durability": "async",
    "index.translog.sync_interval": "30s",
    "index.translog.flush_threshold_size": "248mb"
  }
}

創建mapping es_persist_4

url

post http://ip:port/es_persist_4/_mapping

json

{
  "properties": {
    "servCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "componentHost": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "exceptionCount": {
      "type": "long"
    },
    "sumCallTime": {
      "type": "long"
    },
    "maxCallTime": {
      "type": "long"
    },
    "cAppCode": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    },
    "minCallTime": {
      "type": "long"
    },
    "startTime": {
      "type": "long"
    },
    "endTime": {
      "type": "long"
    },
    "sumFlowSize": {
      "type": "long"
    },
    "totalCount": {
      "type": "long"
    },
    "servVersionProxyType": {
      "type": "text",
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      }
    }
  }
}

es的常用指令

洗掉指定索引，從物理上整個索引的資料洗掉
url

delete http://ip:port/指定的索引名稱

關閉索引，依然占著硬碟，關閉后不可進行io讀寫
url

post http://ip:port/指定的索引名稱/_close

打開索引，占著硬碟，打開后可進行io讀寫，正常使用
url

post http://ip:port/指定的索引名稱/_open

跨集群資料遷移

reindex遷移

b集群請求獲取a集群的資料到b集群里,(b集群組態檔需要加上a集群的白名單，見集群安裝組態檔)
query可以指定想要的資料，下面是獲取指定月份時間段的資料，去掉則是全部資料
“version_type”: "internal"代表覆寫替換沖突的id相同的資料
size是批量條數，太大可能會報錯，太小執行較慢
wait_for_completion=false后臺異步操作

POST http://bip:bport/_reindex?wait_for_completion=false

{
  "source": {
    "index": "a的索引",
    "remote": {
      "host": "http://aip:aport"
    },
    "size": 1000,
    "query": {
      "range": {
        "receiveTime": {
          "gte": 1635696000000,
          "lt": 1638287999000
        }
      }
    }
  },
  "dest": {
    "index": "b的索引",
    "version_type": "internal"
  }
}

reindex取消命令

reindex執行沒結束不想再執行了，成功遷移復制過去的資料依然保留，后續未完成的不再繼續

POST _tasks/node_id:task_id/_cancel

reindex查看進度(可以看到node_id:task_id,任務數等)

GET _tasks?detailed=true&actions=*reindex  

如果遷移自動停止了或者成功了就看不到進度，在資料預覽里可以看到.tasks這個索引，是自動生成存盤task任務的，查看詳情，如果沒有報錯則成功遷移，報錯幾乎原因都是資料太大，所以需要調小size并重試

冷資料備份快照到磁盤

可壓縮成壓縮包剪切到其他機器，減少本集群的硬碟占用，需要在yml指定目錄，見安裝集群組態檔elasticsearch.yml，目錄建議放在硬碟空間大的目錄，防止快照檔案過大導致失敗，
而且指定的目錄要有權限
path.repo: ["/opt/soft/es_backups/backups", “/opt/soft/es_backups/longterm_backups”]

1.創建倉庫：創建一個名稱為my_backup的倉庫

PUT /_snapshot/my_backup

{
  "type": "fs",
  "settings": {
    "location": "/opt/soft/es_backups/backups/my_backup"
  }
}

如果生成快斬訓在/opt/soft/es_backups/backups下生成倉庫my_backup目錄

2.往指定倉庫my_backup保存指定索引b_index快照資料,創建一個名為 snapshot_1 的快照
wait_for_completion=true可設定同步異步，多個索參考逗號隔開

put /_snapshot/my_backup/snapshot_1?wait_for_completion=true

{
  "indices": "b_index"
}

也可以對snapshot_1 的快照增量同步
直接原命令再創建一個快照 snapshot_2 就可以了

3.資料遷出
打包倉庫my_backup,存盤不夠，直接把檔案夾復制走也行

tar -zcvf my_backup.tar.gz my_backup 

可以把壓縮檔案剪切轉儲其他機器,然后洗掉這個檔案夾
做的更徹底些，可以洗掉這個快照和倉庫，每次創建快照都重開一個倉庫，建議用日期后綴

delete /_snapshot/my_backup/snapshot_1
delete /_snapshot/my_backup

資料包檔案遷出后，可以把這個索引洗掉，解除對硬碟的占用

4.資料恢復
如果是新機器上恢復，新機器也要有倉庫目錄
把快照檔案傳過來，在快照目錄下解壓，解壓的檔案會生成目錄my_backup和新機器指定的目錄名稱需要一致
執行
創建倉庫

PUT /_snapshot/my_backup

{
  "type": "fs",
  "settings": {
    "location": "/opt/soft/es_backups/backups/my_backup"
  }
}

讀取資料并創建索引

POST /_snapshot/my_backup/snapshot_1/_restore?wait_for_completion=true

{
  "indices": "index_1"
}

冷資料備份到其他地方，可選

{
  "type": "url",
  "settings": {
    "url": "file:/mount/backups/my_fs_backup_location"
  }
}

Url 引數支持以下協議:

file
ftp
http
https
jar
使用 ftp、 http 或 https 協議的 url 必須使用 repositories.url.allowed _ url 設定顯式地被允許
支持通配符(*)，需要配置引數，上面的組態檔未添加該配置

repositories.url.allowed_urls: ["http://www.example.org/root/*", "https://*.mydomain.com/*?*#*"]

集群安全重啟

關閉前執行

PUT _cluster/settings

{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}

重啟過后執行恢復
請確保每個節點都啟動了，不然會造成主分片負載不均衡

{
    "transient" : {
        "cluster.routing.allocation.enable" : "all"
    }
}

分片移動

若發現主分片負載不均衡，可手動移動分片
將node-1的2號分片移動到node-3

post _cluster/reroute

{
  "commands": [
    {
      "move": {
        "index": "aindex",
        "shard": 2,
        "from_node": "node-1",
        "to_node": "node-3"
      }
    }
  ]
}