為了防止在從 logstash 注入資料時出現資料重復，我正在為帶有 peopleRowId 列的 logstash conf 添加一個 document_id 字串。但是，它沒有得到評估。因此，在我的情況下，我嘗試將檔案 ID 設定為 document_id => "%{[document][projectsRowId]}"，但是由于某種原因并沒有在彈性搜索中評估該 ID，因為我將 ROW_NUMBER() OVER ( ORDER BY a.created_at ) 添加為 projectsRowId 以創建唯一 ID
[
{
"_index" : "projectsv3",
"_type" : "_doc",
"_id" : "%{[document][projectsRowId]}",
"_score" : 1.0,
"_source" : {...single record}
]
我不確定為什么沒有啟用檔案 ID。使用彈性搜索 7 和 ECS 也被禁用。我也嘗試過其他方法，例如帶有指紋的過濾器我也嘗試設定檔案ID，因為document_id => "%{projectsRowId}"在所有情況下它都沒有得到評估
input {
jdbc {
jdbc_driver_library => "C:\\ElasticStack\\mysql-connector-java-8.0.24\\mysql-connector-java-8.0.24.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# mysql jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/corrabla_sercweb"
# The user we wish to execute our statement as
jdbc_user => "root"
jdbc_password => "root"
schedule => "* * * * *"
clean_run => true
# use_column_value => true
# tracking_column => "%{[@metadata][fingerprint]}"
# tracking_column_type => "numeric"
# our query to fetch people details
statement => "select ROW_NUMBER() OVER (
ORDER BY a.created_at
) as projectsRowId , (a.created_at), tr.report_number as 'tech_report_number', tr.file_s3 as 'tech_report_file_name', tr.abstract as 'tech_report_abstract' , c.prefix as 'piPrefix' , c.first_name as 'piFirstName', c.middle_name as 'piMiddleName' ,c.last_name as 'piLastName', b.person_id, d.prefix as 'coPiPrefix' "
# use_column_value => true
# tracking_column => id
# tracking_column_type => "numeric"
}
}
output {
elasticsearch {
action => "create"
hosts => "http://127.0.0.1:9200"
index => "projectsv3"
doc_as_upsert => true
document_id => "%{[document][projectsRowId]}"
}
}
uj5u.com熱心網友回復：
默認情況下，jdbc 輸入會將欄位名稱折疊為小寫，因此您的事件將有一個名為 projectsrowid 的欄位，而不是 projectsRowId。如果您lowercase_column_names => false在輸入上設定，則 `document_id => "%{[projectsRowId]}" 將起作用。
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/428939.html