我打算獲取彈性搜索索引中的所有行，然后將這些行存盤為 CSV 檔案。但是，我嘗試過的大多數方法最終都給了我大小限制錯誤。
curl -k -u username:password -XGET "https://xx.xx.xx.xx:xxxx/foo-index/_search?scroll=10m"
-H 'Content-Type: application/json'
-d'{ "from": 0, "size": 933963, "query" : { "match_all" : {} }, "track_total_hits": true, "_source": ["foo_id"]}'
顯示的錯誤是：
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [933963]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"foo-index","node":"k0OUtLDFRye4gIXGKCKLmQ","reason":{"type":"illegal_argument_exception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [933963]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}}]
問題是我無法減小大小，因為我需要獲取索引中的全部內容。
uj5u.com熱心網友回復：
由于 Elasticsearch 的大小限制為 10k，因此您遇到了例外。您可以使用api 之后的搜索來獲取您將設定的所有檔案，size:10000因此它將是多次呼叫（每次呼叫將獲得 10k 個檔案）以從索引中獲取所有資料。
要使用 PIT，您需要首先使用以下命令生成 PTI ID：
POST /my-index-000001/_pit?keep_alive=1m
API 回傳一個 PIT ID。
{
"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA=="
}
您可以使用 PIT ID，如下面的查詢所示：
GET /_search
{
"size": 10000,
"query": {
"match" : {
"user.id" : "elkbee"
}
},
"pit": {
"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==",
"keep_alive": "1m"
},
"sort": [
{"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos", "numeric_type" : "date_nanos" }}
]
}
上面的查詢將回傳您的前 10k 個檔案，作為回應，您將獲得新的 PIT ID，您可以將其傳遞給相同的查詢以獲取下一組 10k 個檔案。
{
"pit_id" : "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==",
"took" : 17,
"timed_out" : false,
"_shards" : ...,
"hits" : {
"total" : ...,
"max_score" : null,
"hits" : [
...
{
"_index" : "my-index-000001",
"_id" : "FaslK3QBySSL_rrj9zM5",
"_score" : null,
"_source" : ...,
"sort" : [
"2021-05-20T05:30:04.832Z",
4294967298
]
}
]
}
}
PS：沒有解決方案可以在單個 API 呼叫中為您提供超過 10k 的檔案。您需要使用search_after或scroll API。
轉載請註明出處，本文鏈接：https://www.uj5u.com/qukuanlian/415707.html
標籤：