JSON格式的Powershell問題（log4jscanner和utf16）-有解無憂

我正在嘗試從 log4jscanner.exe （用于檢測您的 pc/服務器中是否存在易受攻擊的檔案或組件的 Qualys 軟體）成功檢索一些 JSON 資料，但是在花了很多時間之后，我認為我遇到了 Powershell 的問題。

如果我將以下命令的結果存盤在 Powershell 5.1 中 $a = .\Log4jScanner.exe /scan /report_pretty

結果“顯示”如下：

PS C:\temp> $a |where {$_ -ne ""}
    {
        "scanSummary": {
            "scanEngine": "2.0.2.7",
            "scanHostname": "XXXXXXXXX",
            "scanDate": "2022-01-20T18:02:26 0100",
            "scanDurationSeconds": 28,
            "scanErrorCount": 54,
            "scanStatus": "Partially Successful",
            "scannedFiles": 649020,
            "scannedDirectories": 209514,
            "scannedJARs": 31,
            "scannedWARs": 0,
            "scannedEARs": 0,
            "scannedPARs": 0,
            "scannedTARs": 5,
            "scannedCompressed": 43,
            "vulnerabilitiesFound": 1
        },
        "scanDetails": [
            {
                "file": "XXXXXX.jar",
                "manifestVendor": "Unknown",
                "manifestVersion": "Unknown",
                "detectedLog4j": true,
                "detectedLog4j1x": true,
                "detectedLog4j2x": false,
                "detectedJNDILookupClass": false,
                "detectedLog4jManifest": false,
                "log4jVendor": "log4j",
                "log4jVersion": "1.2.17",
                "cve20214104Mitigated": false,
                "cve202144228Mitigated": true,
                "cve202144832Mitigated": true,
                "cve202145046Mitigated": true,
                "cve202145105Mitigated": true,
                "cveStatus": "Potentially Vulnerable ( CVE-2021-4104: Found )"
            }
        ]
    }

之后，我想將該資料轉換為特定值，首先我嘗試從 json 轉換資料，此處文本變為紅色，并發生以下錯誤：

    PS C:\temp> $a | convertfrom-json
    convertfrom-json : Objet non valide passé, ':' ou '}' attendu. (2): {
    
        "scanSummary": {
    
            "scanEngine": "2.0.2.7",
    
            "scanHostname": "FRBOURWXT013379.vcn.ds.volvo.net",
    
            "scanDate": "2022-01-20T18:02:26 0100",
 .... .... ....

最后，如果我將 $a 的內容復制/粘貼到另一個變數中，例如

$b = '
{
            "scanSummary": {
                "scanEngine": "2.0.2.7",
                "scanHostname": "XXXXXXXXX",
                "scanDate": "2022-01-20T18:02:26 0100",
                "scanDurationSeconds": 28,

... ... ... 
'

這意味著我現在可以訪問轉換后的資料：

PS C:\temp> $b | convertfrom-json

scanSummary
-----------
@{scanEngine=2.0.2.7; scanHostname=XXXXXXXXX; scanDate=2022-01-20T18:02:26 0100; scanDurationSeconds=28; scanErrorCount=54; scanStatus=Partially Successful; scann...

目前 $a 型別是 Object[] ， $b 型別是 String。

所以我嘗試將 $a 轉換為字串

PS C:\temp> $a = [string] $a
PS C:\temp> $a
{      "scanSummary": {          "scanEngine": "2.0.2.7",          "scanHostname": "XXXXXXXXX",          "scanDate": "2022-01-20T18:02:26 0100",          "scanDurationSeconds": 28,          "scanErrorCount": 54,          "scanStatus": "Partially Successful",          "scannedFiles": 649020,          "scannedDirectories": 209514,          "scannedJARs": 31,          "scannedWARs": 0,          "scannedEARs": 0,          "scannedPARs": 0,          "scannedTARs": 5,          "scannedCompressed": 43,          "vulnerabilitiesFound": 1      },      "scanDetails": [          {              "file": "XXXXX.jar",              "manifestVendor": "Unknown",              "manifestVersion": "Unknown",              "detectedLog4j": true,              "detectedLog4j1x": true,              "detectedLog4j2x": false,              "detectedJNDILookupClass": false,              "detectedLog4jManifest": false,              "log4jVendor": "log4j",              "log4jVersion": "1.2.17",              "cve20214104Mitigated": false,              "cve202144228Mitigated": true,              "cve202144832Mitigated": true,              "cve202145046Mitigated": true,              "cve202145105Mitigated": true,              "cveStatus": "Potentially Vulnerable ( CVE-2021-4104: Found )"          }      ]  }

然后從json轉換它，但它是一團糟

PS C:\temp> $a | convertfrom-json
convertfrom-json : Objet non valide passé, ':' ou '}' attendu. (2): {      "scanSummary": {          "scanEngine": "2.0.2.7",
       "scanHostname": "XXXXXX",          "scanDate":
"2022-01-20T18:02:26 0100",          "scanDurationSeconds": 28,          "scanErrorCount":
54,          "scanStatus": "Partially Successful",          "scannedFiles": 649020,
"scannedDirectories": 209514,          "scannedJARs": 31,          "scannedWARs": 0,

Finally, if i export any data to .json file, i can't open it with notepad or codium (every character = nul nul nul nul) whereas i can access it with get-content within powershell.

It seems there's some hidden characters or i don't know what, but i can't handle how to easily convert and access json data in my case.

Is there anything missing ?

Thanks a lot for your support guys !

EDIT 1 - if i save the output, i can't open the .json file correctly ,but Powershell seems to understand it well : JSON 格式的 Powershell 問題（log4jscanner 和 utf16）

uj5u.com熱心網友回復：

Log4jScanner.exe 輸出 Unicode。

PowerShell中存在一個錯誤，導致將 Unicode 位元組發送到其 STDOUT/STDERR 流的程式的輸出被破壞。

很容易確認 - 當您運行命令時

Log4jScanner.exe /scan_directory C:\something /report_pretty > output.json

在中cmd.exe，output.json則將是整潔的 UTF-16：

0d 00 0a 00 7b 00 0d 00 0a 00 20 00 20 00 20 00 .?.?{?.?.? ? ? ?
20 00 22 00 73 00 63 00 61 00 6e 00 53 00 75 00 ?"?s?c?a?n?S?u?
6d 00 6d 00 61 00 72 00 79 00 22 00 3a 00 20 00 m?m?a?r?y?"?:??

但是 PowerShell 會盲目地假設程式的輸出流采用單位元組編碼，并再次將其編碼為 UTF-16 ，包括實際上屬于 UTF-16 字符的 NUL 位元組：

ff fe 0d 00 0a 00 00 00 0d 00 0a 00 00 00 7b 00 ?t.?.???.?.???{?
00 00 0d 00 0a 00 00 00 0d 00 0a 00 00 00 20 00 ??.?.???.?.??? ?
00 00 20 00 00 00 20 00 00 00 20 00 00 00 22 00 ?? ??? ??? ???"?

在這里，我們看到 UTF-16 BOM ( ff fe)，然后在原始輸出中存在 NUL 的每個位置插入一個真正的NUL 字符00 00，換行符除外，這就是我們仍然看到常規\r\n( 0d 00 0a 00) 的原因。例如，空格（20 00在 UTF-16 中）將變為20 00 00 00，并在文本編輯器中顯示為空格加 NUL，正如您在 Notepad 中看到的那樣。

這當然是可怕的。

您的選擇是：

Log4jScanner.exe從運行cmd.exe
在決議之前從輸出中洗掉多余的 NUL 字符

后者會是這樣的：

$json = Log4jScanner.exe /scan_directory C:\something /report_pretty
$data = $json.Replace(([char]0).ToString(), "") | ConvertFrom-Json

.NET 字串可以合法地包含 NUL 字符（例如 C 字串不能），但是我們期望程式的 JSON 輸出中沒有合法的 NUL 字符，這就是為什么將它們全部扔掉的原因，但它肯定不漂亮 - 并且它僅適用于實際上不包含 Unicode 字符的程式輸出（這里恰好是這種情況，JSON 中的所有字符都在 ASCII 范圍內）。

uj5u.com熱心網友回復：

這是我發現的一些奇怪的代碼，可以識別 unicode (utf16-le) no bom 檔案。這偶爾會出現在視窗中。記事本也可以識別它（并且 utf8 no bom）。

# istextunicode.ps1

param([Parameter(ValueFromPipeline=$True)] $filename)

# https://devblogs.microsoft.com/scripting/use-powershell-to-interact-with-the-windows-api-part-1/

begin {

$MethodDefinition = @'
[DllImport("Advapi32",SetLastError=false)]
public static extern bool IsTextUnicode(byte[] buf, int len, 
  ref IsTextUnicodeFlags opt);

[Flags]
public enum IsTextUnicodeFlags:int 
{
  IS_TEXT_UNICODE_ASCII16            = 0x0001,
  IS_TEXT_UNICODE_REVERSE_ASCII16    = 0x0010,
  IS_TEXT_UNICODE_STATISTICS         = 0x0002,
  IS_TEXT_UNICODE_REVERSE_STATISTICS = 0x0020,
  IS_TEXT_UNICODE_CONTROLS           = 0x0004,
  IS_TEXT_UNICODE_REVERSE_CONTROLS   = 0x0040,
  IS_TEXT_UNICODE_SIGNATURE          = 0x0008,
  IS_TEXT_UNICODE_REVERSE_SIGNATURE  = 0x0080,
  IS_TEXT_UNICODE_ILLEGAL_CHARS      = 0x0100,
  IS_TEXT_UNICODE_ODD_LENGTH         = 0x0200,
  IS_TEXT_UNICODE_DBCS_LEADBYTE      = 0x0400,
  IS_TEXT_UNICODE_NULL_BYTES         = 0x1000,
  IS_TEXT_UNICODE_UNICODE_MASK       = 0x000F,
  IS_TEXT_UNICODE_REVERSE_MASK       = 0x00F0,
  IS_TEXT_UNICODE_NOT_UNICODE_MASK   = 0x0F00,
  IS_TEXT_UNICODE_NOT_ASCII_MASK     = 0xF000
}
'@

  Add-Type Advapi32 $MethodDefinition -Namespace Win32
  $totalcount = 8
}

process {
  if ( (get-item $filename).length -lt 1mb ) {
  
    #$bytes = [io.file]::ReadAllBytes($filename)
    $bytes = get-content $filename -encoding byte -totalcount $totalcount
    
    # reset every time
    [Win32.Advapi32 IsTextUnicodeFlags]$opt = 0xffff
    
    $result = [win32.advapi32]::IsTextUnicode($bytes, $bytes.length, [ref]$opt)
    #$result = [win32.advapi32]::IsTextUnicode($bytes, $totalcount, [ref]$opt)
      
    [pscustomobject]@{
      Filename = $filename
      Result = $result
      Flags = $opt
    }
    #if($result) { write-host $filename }
  }
}

# error.log
# icacls (no bom), task scheduler, regedit
# gpreport.html

# $a = ls -force -r -file -exclude *.dll,*.exe,*.mui,*.jpg,*.jar,*.zip,*.msb,*.dat | get-item | istextunicode | where result
# -filter *.ini *.txt *.log

# $a = ls  -force -Recurse -Filter *.ini | get-item | istextunicode | where {$_.result -and $_.flags -notmatch 'signature' }

在 cmd 中作為管理員執行（在 powershell 中它添加了錯誤的編碼 bom 簽名）：

sfc > file

然后在powershell中：

.\istextunicode file

Filename Result                                                                                                     Flags
-------- ------                                                                                                     -----
file       True IS_TEXT_UNICODE_ASCII16, IS_TEXT_UNICODE_STATISTICS, IS_TEXT_UNICODE_CONTROLS, IS_TEXT_UNICODE_NULL_BYTES

format-hex file | select -first 1


           Path: C:\users\admin\foo\file

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   0D 00 0D 00 0A 00 4D 00 69 00 63 00 72 00 6F 00  ......M.i.c.r.o.

轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/418468.html

標籤：

上一篇：為什么Powershell命令替換對我不起作用

下一篇：Powershell過濾名稱串列