如何保留第一行和最后一行并在日志檔案中洗掉

我正在運行幾個火花作業，當作業正在等待資源時，它們會產生如下日志。

22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:27 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:29 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:31 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:33 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:35 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:37 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again

作業完成后，我想通過執行一些 perl 命令洗掉多余的 msgs 來減少日志檔案的大小。我想要像下面這樣的輸出，只有第一行和最后一行。

22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again

我嘗試了類似下面的方法，但由于時間戳正在更改，我無法使用貪婪運算子。

perl -0777 -ne ' { s/(^\d\d\/\d\d\/\d\d \d\d:. ? Could not get any http protocol, using HTTP and will try to get protocol again.) /$1/mg;print } ' log-file.

在實際場景中，日志部分可能會重復多次。像這樣的東西。

sometext1
sometext2
22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:27 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:29 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:31 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:33 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:35 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:37 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext3
sometext4
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:53 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext5
sometext6

這可以使用正則運算式解決嗎？

所需輸出：

sometext1
sometext2
22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext3
sometext4
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:53 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext5
sometext6

uj5u.com熱心網友回復：

你可以試試這個正則運算式：

^(\d{2}(?:\/\d{2}){2} \d{2}(?::\d{2}){2}(.*$)\s*)(\d{2}(?:\/\d{2}){2} \d{2}(?::\d{2}){2}\2\s*) 

點擊演示

解釋：

• ^ - 匹配一行的開頭
• (\d{2}(?:\/\d{2}){2} \d{2}(?::\d{2}){2}(.*$)\s*)- 第一個日志行存盤為組 1
  • (\d{2}(?:\/\d{2}){2} \d{2}(?::\d{2}){2}- 匹配數字XX/XX/XX XX:XX:XX的格式模式X
  • (.*$)- 匹配所有內容，直到行尾。任何匹配的內容都存盤在Group 2中。實際日志（沒有時間戳）存盤在該組中。
  • \s* - 匹配 0 個或多個空格
• (\d{2}(?:\/\d{2}){2} \d{2}(?::\d{2}){2}\2\s*) - 匹配所有剩余的連續日志行，以格式XX/XX/XX XX:XX:XX開頭，后跟第 2 組的內容，但只有最后一個這樣的日志行將存盤在第 3 組中

現在，將每個匹配項替換為第 1 組的內容，然后是第 3 組 $1$3

uj5u.com熱心網友回復：

雖然使用正則運算式是可能的，但這可以通過普通的 Perl 代碼輕松解決。我認為代碼更清晰，更易于維護。我在您的示例輸入中添加了 3 行，以測驗我們在與搜索匹配的行上結束的邊緣情況。

use strict;
use warnings;

# This string can be replaced as needed
my $str = "INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again";

my ($first, $last);

while (<DATA>) {
    if (/\Q$str/) {                      # if pattern matches current line
        if ($first) {                    # if this is an "in between" line
            $last = $_;                  # save line and go next
        } else {                         # if this is the first line
            print if not eof;            # print it..
            $first = $_;                 # ...save line and go next
        }
        print if eof;                    # print last line to avoid edge cases
    } elsif ($first && $last) {          # $str didn't match: finished a range of lines
        print $last, $_;                 # print and reset
        $first = undef;
        $last = undef;
    } else {
        print;                           # print everything else
    }
}

__DATA__
sometext1
sometext2
22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:27 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:29 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:31 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:33 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:35 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:37 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext3
sometext4
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:53 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext5
sometext6
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again

輸出：

sometext1
sometext2
22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext3
sometext4
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:53 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext5
sometext6
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:51 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again

uj5u.com熱心網友回復：

正則運算式解決方案

use warnings;
use strict;
use feature 'say';

die "Usage: $0 file\n" if not @ARGV;

my $fc = do { local $/; <> };  # file content

my $d2 = qr{[0-9]{2}};
my $ts = qr{$d2/$d2/$d2 $d2:$d2:$d2};

$fc =~ s{ ($ts\s*(.*?)\n) (?: $ts\s*\g{-1}\n )  ( $ts\s*\g{-2}\n ) }{$1$3}gx;

say $fc;

它匹配一個時間戳，然后是該行的其余部分。然后，一個非捕獲組時間戳 上次捕獲的內容^?盡可能多次匹配到另一個這樣的模式，因為我們需要第一個和最后一個時間戳不同。它在任何其他文本處停止。然后它重復，由/g。

保留每組中第一行和最后一行的另一種方法是在非捕獲組中捕獲，并在替換中使用它，因為它將是行組中的最后一個這樣的模式

$fc =~ s{ ($ts\s*(.*?)\n) (?: ($ts\s*\g{2}\n) )  }{$1$3}gx; 

現在我們需要計算我們的反向參考\g{2}，而不是使用相對參考。

上述兩種變體都使用給定的輸入產生所需的輸出

sometext1
sometext2
22/01/03 14:42:25 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:39 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext3
sometext4
22/01/03 14:42:49 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
22/01/03 14:42:53 INFO rpc.b: Could not get any http protocol, using HTTP and will try to get protocol again
sometext5
sometext6

除了TLP 的答案（感謝 TLP）中提供的那個之外，這沒有針對可能的奇怪輸入或邊緣情況進行測驗，但我不希望有驚喜。如果有意外行為，請報告。

如果出于某種原因需要，可以將其壓縮為單線。

^? 通過相對反向參考，\g{-1}

標籤：

上一篇：索引perl陣列

下一篇：使用cURL訪問和下載帶有動態cookie的檔案