我有一個 txt 檔案，其中包含超過 100,000 行的標題列欄位，這些欄位由制表符分隔，并且包含重復的條目。現在，當 VATRegistrationNumber 重復時，我想通過使用批處理腳本或 PowerShell 腳本來洗掉那些重復的條目（當 GroupID 1000 的條目已經存在時，GroupID 2000 再次出現）。
輸入檔案：
Supplier GroupID Name1 City VATRegistrationNumber Region
9032512 1000 PENRITH PERFORMING AND VISUAL ARTS Penrith 97003605089 NSW
9032923 1000 TORONTO MULTI PURPOSE CENTRE INC Toronto 85659352794 NSW
9035991 1000 SYDNEY'S BEST MAINTENANCE AND CLEAN Panania 42616361365 NSW
9044080 1000 FIRST PEOPLES DISABILITY NETWORK Sydney 58169154330 NSW
9044292 1000 PUNCHY DIGITAL MEDIA PTY LTD South Melbourne 22166978669 VIC
9044691 1000 POLARON LANGUAGE SERVICES PTY LTD St Kilda 99120446362 VIC
9047594 1000 WORKSPACE COMMERCIAL FURNITURE PTY Rosebery 16619275213 NSW
9075591 1000 ENGADINE COMMUNITY SERVICES INC Engadine 39108386726 NSW
9076178 1000 CENTRAL COAST AGEING AND DISABILITY Newcastle West 60874001192 NSW
9076319 1000 YASS SOLDIERS' CLUB LTD Yass 44000988152 NSW
9076370 1000 ARTFUL FOOD CO. PTY LIMITED EMU PLAINS 52600861612 NSW
9078232 1000 WSG AUSTRALIA PTY LTD NORWOOD SOUTH 76604823430 SA
9078312 1000 CAZNER PTY LTD Manly 39648003570 NSW
9078463 1000 MILSONS POINT DIGITAL PTY LTD Milsons Point 88635600681 NSW
9079757 1000 RISK & SECURITY MANAGEMENT PTY LTD Brisbane 71613136048 QLD
9079793 1000 PETRUSIC, MILOJKA Seven Hills 86077460484 NSW
8487483 2000 FOTI, STEPHANIE BLACKTOWN 78620350307 NSW
7458490 2000 STAFF CHECK PTY LIMITED Marrickville 36227046572 NSW
75637 2000 ANDERSON, ALLAN GRAHAM Gladesville 49340951752 NSW
362789 2000 STAR TRACK EXPRESS PTY LIMITED CRONULLA 82104931562 NSW
952678 2000 CALCUTTA GROUP PTY LTD KATOOMBA 52699520223 NSW
2403417 2000 The trustee for Moore Family Trust NORWOOD SOUTH 76604823430 SA
94905 2000 MEYER PTY LTD Manly 39648003570 NSW
426783 2000 DAWSON DIGITAL PTY LTD Milsons Point 88635600681 NSW
8588214 2000 HOME MANAGEMENT PTY LTD Brisbane 71613136048 QLD
預期的輸出檔案：
Supplier GroupID Name1 City VATRegistrationNumber Region
9032512 1000 PENRITH PERFORMING AND VISUAL ARTS Penrith 97003605089 NSW
9032923 1000 TORONTO MULTI PURPOSE CENTRE INC Toronto 85659352794 NSW
9035991 1000 SYDNEY'S BEST MAINTENANCE AND CLEAN Panania 42616361365 NSW
9044080 1000 FIRST PEOPLES DISABILITY NETWORK Sydney 58169154330 NSW
9044292 1000 PUNCHY DIGITAL MEDIA PTY LTD South Melbourne 22166978669 VIC
9044691 1000 POLARON LANGUAGE SERVICES PTY LTD St Kilda 99120446362 VIC
9047594 1000 WORKSPACE COMMERCIAL FURNITURE PTY Rosebery 16619275213 NSW
9075591 1000 ENGADINE COMMUNITY SERVICES INC Engadine 39108386726 NSW
9076178 1000 CENTRAL COAST AGEING AND DISABILITY Newcastle West 60874001192 NSW
9076319 1000 YASS SOLDIERS' CLUB LTD Yass 44000988152 NSW
9076370 1000 ARTFUL FOOD CO. PTY LIMITED EMU PLAINS 52600861612 NSW
9078232 1000 WSG AUSTRALIA PTY LTD NORWOOD SOUTH 76604823430 SA
9078312 1000 CAZNER PTY LTD Manly 39648003570 NSW
9078463 1000 MILSONS POINT DIGITAL PTY LTD Milsons Point 88635600681 NSW
9079757 1000 RISK & SECURITY MANAGEMENT PTY LTD Brisbane 71613136048 QLD
9079793 1000 PETRUSIC, MILOJKA Seven Hills 86077460484 NSW
8487483 2000 FOTI, STEPHANIE BLACKTOWN 78620350307 NSW
7458490 2000 STAFF CHECK PTY LIMITED Marrickville 36227046572 NSW
75637 2000 ANDERSON, ALLAN GRAHAM Gladesville 49340951752 NSW
362789 2000 STAR TRACK EXPRESS PTY LIMITED CRONULLA 82104931562 NSW
952678 2000 CALCUTTA GROUP PTY LTD KATOOMBA 52699520223 NSW
這是我到目前為止在 PowerShell 中的內容，但它沒有提供所需的結果，因為洗掉的行比我預期的要多。
$Unique = [System.Collections.Generic.HashSet[string]]::new()
Get-Content C:\Test\Input.txt |ForEach-Object {
if ($Unique.Add(($_.Split('VATRegistrationNumber'))[-2])) { $_ }
} | Set-Content C:\Test\Output.txt
任何幫助將不勝感激。謝謝
更新：
謝謝大家的幫助。我個人發現 iRon 提供的解決方案最有幫助。我只需要進行一些調整以適應我的要求。這是我現在使用的最終 PowerShell 代碼，在包含超過 120,000 個條目的檔案上運行需要不到 1 分鐘的時間：
$OutputTXT = ".\Output.txt"
$Unique = [System.Collections.Generic.HashSet[string]]::new()
Import-Csv .\Input.txt -Delimiter "`t" |ForEach-Object {
if ($Unique.Add($_.VATRegistrationNumber)) { $_ }
} |
ConvertTo-Csv -Delimiter "`t" -NoTypeInformation |
ForEach-Object { $_ -replace '"' } |
Set-Content $OutputTXT -Encoding Unicode
使用 Export-Csv 時，所有輸出欄位都包含雙引號。所以，我做了一些研究，現在改用 ConvertTo-Csv，因為它允許我通過使用 ForEach-Object { $_ -replace '"' } 來消除雙引號。此外，輸出檔案中的第一行包含型別我不想擁有并通過使用 -NoTypeInformation 擺脫的資訊。
boxdog 提供的解決方案也非常好用，速度超級快：
$OutputTXT = ".\Output.txt"
Import-Csv 'input.txt' -Delimiter `t |
Group-Object -Property VATRegistrationNumber |
ForEach-Object {
if($_.Count -gt 1) {
$_.Group | Select-Object -First 1
}
else {
$_.Group
}
} |
ConvertTo-Csv -Delimiter "`t" -NoTypeInformation |
ForEach-Object { $_ -replace '"' } |
Set-Content $OutputTXT -Encoding Unicode
uj5u.com熱心網友回復：
知道#11221Select-Object -Unique 是不必要的緩慢和詳盡的，我確實會為此使用Hashset，但我建議您簡單地使用/ cmdlet 而不是Get-Content/ cmdlet，因為它們會自動處理您的屬性（列）：Set-ContentImport-CsvExport-Csv
$Unique = [System.Collections.Generic.HashSet[string]]::new()
Import-Csv .\Input.txt -Delimiter "`t" |ForEach-Object {
if ($Unique.Add($_.VATRegistrationNumber)) { $_ }
} |Export-Csv .\Output.txt -Delimiter "`t"
uj5u.com熱心網友回復：
這適用于給定的示例，但我不確定 100,000 個條目的性能如何。試一試，看看它是否有幫助。
Import-Csv 'input.txt' -Delimiter `t |
Group-Object -Property VATRegistrationNumber |
ForEach-Object {
if($_.Count -gt 1) {
$_.Group | Select-Object -First 1
}
else {
$_.Group
}
}
uj5u.com熱心網友回復：
@ECHO OFF
SETLOCAL enabledelayedexpansion
rem The following settings for the source directory, destination directory,
rem and filenames are names
rem that I use for testing and deliberately include names which include spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
SET "destdir=u:\your results"
SET "filename1=%sourcedir%\q72613575.txt"
SET "outfile=
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/491634.html
下一篇：將區域變數用于“回傳”結果的函式