std::wcin.eof()、UTF-8和不同系統上的語言環境

我對 C 流及其對 Unicode 的處理知之甚少，試圖了解為什么其他人撰寫的代碼會以這種方式運行。如果有人能向我解釋發生了什么，我將不勝感激。

MCVE :

#include <string>
#include <iostream>

int main() {
  std::basic_string<wchar_t> line;
  std::locale::global(std::locale("")); // This
  std::wcout.imbue(std::locale(""));    // This
  std::wcin.imbue(std::locale(""));     // This
  for (;;) {
    std::getline(std::wcin, line);
    if (std::wcin.eof()) {
      std::wcout << L"EOF" << std::endl;
      break;
    }
    std::wcout << line << std::endl;
  }
}

樣本輸入test.txt：

( ) ライン
second line

編輯：十六進制轉儲test.txt：

$ xxd test.txt
00000000: 2820 2920 e383 a9e3 82a4 e383 b30a 7365  ( ) ..........se
00000010: 636f 6e64 206c 696e 650a                 cond line.

結果

在 CentOS 服務器上，這是結果 (1)：

$ ./a.out < test.txt
( ) ライン
second line
EOF

在我的 Mac 上 (2)：

$ ./a.out < test.txt
( )  EOF

如果我注釋掉三個標記的語言環境行，Redhat 輸出 (3)：

$ ./a.out < test.txt
EOF

而 Mac 輸出 (4)：

$ ./a.out < test.txt
( ) ライン
second line
EOF

問題

• 為什么第二個 (2) 結果檢測到 EOF 中線？之前的第二個空格EOF是哪里來的？（這個結果最讓我困惑。）
• 為什么第三 (3) 個結果會立即檢測到 EOF？
• 最重要的是：如何始終始終如一地獲得第一個 (1) 或最后一個結果 (4)？

環境

這是兩臺機器的環境：

CentOS Linux release 7.5.1804 (Core):

$ c   --version
c   (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

macOS Big Sur (version 11.6):

$ c   --version
Apple clang version 12.0.5 (clang-1205.0.22.11)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

Bonus

One additional puzzle. If I change the input to this (i.e. just add two more spaces inside the parentheses):

(   ) ライン
second line

the original (uncommented) code outputs this on Mac:

$ ./a.out < test.txt
(   )   ララライ翕翕ン
second line
EOF

Those are not artifacts of a messed-up terminal; all those extra characters are actually there:

$ ./a.out < test.txt | xxd
00000000: 2820 2020 2920 2020 e383 a9e3 83a9 e383  (   )   ........
00000010: a9e3 82a4 e7bf b7e7 bfb7 e383 b30a 7365  ..............se
00000020: 636f 6e64 206c 696e 650a 454f 460a       cond line.EOF.

Like... what?

EDIT In response to Giacomo Catenazzi's comment, I changed EOF printing from char to wide, which did fix one weirdness regarding input. My core issue is with reading wcin though, which proves to be unrelated.

EDIT Difference between std::getline and std::wcin.get

Here is the data obtained by getline. In this case, I don't get EOF, but the data is still weird:

std::wcout.imbue(std::locale("C")); // prevent commas
for (;;) {
  std::getline(std::wcin, line);
  if (std::wcin.eof()) {
    std::wcout << L"EOF" << std::endl;
    break;
  }
  int i, l = line.length();
  for (i = 0; i < l; i  ) {
    wchar_t ch = line.at(i);
    std::wcout << std::hex << (int) ch << L" ";
  }
  std::wcout << std::endl;
}

Output:

28 20 29 20 20 0 30e9 30e9 30e9 30a4 30a4 30a4 30f3 
73 65 63 6f 6e 64 20 6c 69 6e 65 
EOF

Where does the 0 come from? What's with the repeated characters? The characters following the 0 translate to ララライイイン. (Note that here I do not try to output the received characters to wcout, only the numeric values, in order to eliminate any possible effects of output encoding.)

The data obtained by get is different, but no less strange:

// ...
std::wcout.imbue(std::locale("C")); // prevent commas
for (;;) {
  wchar_t ch = std::wcin.get();
  if (std::wcin.eof()) {
    std::wcout << L"EOF" << std::endl;
    break;
  }
  std::wcout << std::hex << (int) ch << L" ";
  if (std::char_traits<wchar_t>::eq(ch, std::wcin.widen('\n'))) {
    std::wcout << std::endl;
  }
}

Output:

28 20 29 20 7ffe 7ffe 30e9 7ffe 7ffe 30a4 7ffe 7ffe 30f3 a 
73 65 63 6f 6e 64 20 6c 69 6e 65 a 
EOF

This translates to 翾翾ラ翾翾イ翾翾ン. Where do those 7ffe characters come from?

uj5u.com熱心網友回復：

這是一個libc 錯誤。

請注意錯誤報告說它只影響std::wcin而不是檔案流，但在我的實驗中情況并非如此。所有wchar_t流似乎都受到影響。

另一個主要的開源實作 libstdc 沒有這個錯誤。通過針對 libstdc 構建整個應用程式（包括所有動態庫，如果有），可以避免 libc 錯誤。

如果這不是一個選項，那么處理該錯誤的一種方法是使用窄char流，然后在需要時將字符（可能以 UTF-8 編碼到達wchar_t）分別重新編碼為（可能是 UCS-4）。另一種方法是wchar_t完全擺脫并在整個程式中使用 UTF-8，從長遠來看這可能會更好。

標籤：c macos locale

上一篇：模擬kCGEventOtherMouseDown僅適用于右鍵單擊

下一篇：為什么Pakku在MacOSBigSur上使用時會因“無法找到本機庫libarchive.13.dylib”而崩潰