我有多個要合并的 variableStep 格式的 WIG 檔案。這些檔案用于在基因組背景關系中存盤堿基對 - 分數對。
例如檔案 A 和 B 可能如下所示：
一種
variableStep chrom=chr1
9967151 1.0
9967152 0.5
variableStep chrom=chr2
1107150 0.2
1107151 0.3
乙
variableStep chrom=chr1
10001000 0.3
10001001 0.4
variableStep chrom=chr2
20002000 0.9
20002001 0.7
我正在尋找的最終結果是：
AxB
variableStep chrom=chr1
9967151 1.0
9967152 0.5
10001000 0.3
10001001 0.4
variableStep chrom=chr2
1107150 0.2
1107151 0.3
20002000 0.9
20002001 0.7
你知道我如何有效地合并兩個這樣的檔案的有效版本嗎？
它們可能會變得相當大，我想避免回圈遍歷它并過多地跟蹤標題行。例如，我正在尋找 bash 命令或 R 中存在的東西。
我目前擁有的大型最終檔案的一小部分的大小約為 800 MB，但隨著時間的推移，這將急劇增長到數十到數百 GB 的范圍。
最好對每個塊進行數字排序，但能夠使用它并不重要。
The file will in a later step be transformed into a smaller binary file in the so called BigWIG format.
Background:
I want to build one large file to store information for all base pairs in the human genome with all chromosomes in one file. Since my scores that I store will be calculated in smaller individual files over time I want to update the final large file each time I get a new snippet like A or B. Ideally, these should even be unique, so that lines should be integrated only when they are not yet available, but for now the simple merge would be enough
What I tried so far:
I know how to use cat to concatenate files but the problem here is that I will have more header lines than it should have
Also, I have a solution where I loop through each line, keep track of which chromosome I currently have and append the lines at the correct position. However, this takes rather long for a large dataset
Using rtracklayer::import in R, I can read the WIG files, append them, make them unique and export them again. This works, but it is very memory consuming to read the whole "database" file into memory each time I want to update
Thank you!
uj5u.com熱心網友回復：
假設：
- 無需（重新）排序資料
- “小”檔案可以放入記憶體
- 保留重復分數
設定：
$ cat smallfile
variableStep chrom=chr1 # merge
10001000 0.3
10001001 0.4
variableStep chrom=chr2 # merge
1107150 0.2 # duplicate
20002000 0.9
20002001 0.7
variableStep chrom=chr78 # new
3341000 0.3
3526001 0.4
$ cat bigfile
variableStep chrom=chr1
9967151 1.0
9967152 0.5
variableStep chrom=chr117
2347150 0.2
2347151 0.3
variableStep chrom=chr2
1107150 0.2
1107151 0.3
variableStep chrom=chr36
4727150 0.2
4727151 0.3
一個GNU awk（用于多維陣列）的想法：
awk '
### 1st file / smallfile
FNR==NR { if ($1 == "variableStep") { # if new header line then reset variables ...
chrom=$2
n=0
next
}
scores[chrom][ n]=$0 # store non-header lines in scores[] array
next
}
### 2nd file / bigfile
{ print # print current line
if ($1 == "variableStep") # if current line is a header line and ...
if ($2 in scores) { # the "chrom=chrX" is an index in the scores[] array then ...
for (n in scores[$2]) # loop through 2nd index and ...
print scores[$2][n] # print the array entry to stdout (ie, add to our output stream)
delete scores[$2] # delete data to keep from being added again in the "END" block
}
}
### add "new" entries (from 1st file / smallfile)
END { for (chrom in scores) {
print "variableStep", chrom
for (n in scores[chrom])
print scores[chrom][n]
}
}
' smallfile bigfile > newbigfile
這會產生：
$ cat newbigfile
variableStep chrom=chr1 # merged
10001000 0.3 # added
10001001 0.4 # added
9967151 1.0
9967152 0.5
variableStep chrom=chr117
2347150 0.2
2347151 0.3
variableStep chrom=chr2 # merged
1107150 0.2 # added; duplicate
20002000 0.9 # added
20002001 0.7 # added
1107150 0.2 # duplicate
1107151 0.3
variableStep chrom=chr36
4727150 0.2
4727151 0.3
variableStep chrom=chr78 # added; new
3341000 0.3 # added
3526001 0.4 # added
筆記：
- 可以添加排序，但會增加記憶體使用量（如果只是在給定標題行中對分數進行排序，則可能不是問題）
- 洗掉重復分數是可行的，記憶體使用量可以忽略不計
- 在“標題行”排序順序中添加“新”條目
bigfile是可行的，需要更多編碼
轉載請註明出處，本文鏈接：https://www.uj5u.com/qiye/445482.html