前言
今天這篇博客是接我的上一篇博客 https://www.cnblogs.com/y-yp/p/12191258.html,繼續介紹一下MMF在Linux上的用法
ps:本來本地除錯完case,想放到服務器上跑跑看,結果竟然報"PlatformNotSupportedException",然后仔細一查,竟然發現MMF在Windows和Linux上的用法不一樣,,,"mapName"引數僅作為Window平臺的一個特性,在Linux平臺上只能傳"null",于是就有了今天這篇博客
實作
既然“mapName”不能使用,經過測驗選定使用了FileStream的這個多載
具體細節就不在介紹了,有疑問的話可以參考我的上一篇博客 https://www.cnblogs.com/y-yp/p/12191258.html,這里直接給實作
先定義“行資料資訊記錄“,這個用來讀取資料的時候用,一行資料只生成一條記錄,所以在大資料量的情況下也不會占用很多記憶體
public class RowInfo
{
/// <summary>
/// 行資料體積(單位位元組)
/// </summary>
public long Capacity { get; set; }
/// <summary>
/// 行單元格個數
/// </summary>
public int CellQuantity { get; set; }
}
然后開始將資料寫入MMF檔案,并獲取到”行資料資訊記錄“
//準備資料
var data = https://www.cnblogs.com/augustuss/p/new List();
for (var i = 0; i < 100; i++)
{
var rowData = new string[100];
for (var j = 0; j < 100; j++)
{
rowData[j] = $"{i}-{j}";
}
data.Add(rowData);
}
//統計mmf檔案體積,包含單元格資料的體積Encoding.UTF8.GetBytes(x).Length和默認單元格資料長度int型別占4位元組
var mmfCapacity = data.Sum(x => x.Sum(x => Encoding.UTF8.GetBytes(x).Length + 4));
var path = Environment.CurrentDirectory + "\\" + "test.txt";
using var writerFs = new FileStream(path, FileMode.Create, FileAccess.ReadWrite);
using var writerMMF = MemoryMappedFile.CreateFromFile(writerFs, null, mmfCapacity, MemoryMappedFileAccess.ReadWrite, HandleInheritability.Inheritable, true);
//記錄行資料資訊
var rowInfos = new List<RowInfo>();
var totalWriterOffset = 0;
foreach (var rowData in data)
{
var rowBuffers = rowData.Select(x => Encoding.UTF8.GetBytes(x)).ToList();
//計算行資料總體積
var capacity = rowBuffers.Sum(x => x.Length + 4);
//根據當前偏移和需要讀取的長度創建accessor
using var accessor = writerMMF.CreateViewAccessor(totalWriterOffset, capacity);
//統計同行內單元格偏移
var offset = 0L;
foreach (var cellBuffer in rowBuffers)
{
if (cellBuffer.Length > 0)
{
var dataSize = cellBuffer.Length;
accessor.Write(offset, dataSize);
accessor.WriteArray(offset + 4, cellBuffer, 0, dataSize);
offset += 4 + dataSize;
}
else
{
accessor.Write(offset, 0);
offset += 4;
}
}
//記錄行資料資訊
var rowInfo = new RowInfo()
{
Capacity = capacity,
CellQuantity = rowBuffers.Count()
};
rowInfos.Add(rowInfo);
//總位移向前走一行資料的長度
totalWriterOffset += capacity;
}
return rowInfos;
通過”行資料資訊記錄“還原資料,這里可以將讀取出來的資料寫入自己的excel或者是csv檔案,不再贅述
var result = new List<string[]>();
var path = Environment.CurrentDirectory + "\\" + "test.txt";
//從行資料資訊記錄統計mmf檔案總體積
var mmfCapacity = rowInfos.Sum(x => x.Capacity);
var totalReaderOffset = 0;
using var readerFs = new FileStream(path, FileMode.Open, FileAccess.ReadWrite);
using var readerMMF = MemoryMappedFile.CreateFromFile(readerFs, null, mmfCapacity, MemoryMappedFileAccess.ReadWrite, HandleInheritability.Inheritable, true);
foreach(var rowInfo in rowInfos)
{
var rowData = https://www.cnblogs.com/augustuss/p/new string[rowInfo.CellQuantity];
using var accessor = readerMMF.CreateViewAccessor(totalReaderOffset, rowInfo.Capacity);
var position = 0;
for (int cellIndex = 0; cellIndex < rowInfo.CellQuantity; cellIndex++)
{
var cellSize = accessor.ReadInt32(position);
var buffer = new byte[cellSize];
accessor.ReadArray(position + 4, buffer, 0, cellSize);
rowData[cellIndex] = Encoding.UTF8.GetString(buffer);
position += 4 + cellSize;
}
result.Add(rowData);
}
if (File.Exists(path))
{
File.Delete(path);
}
考慮使用記憶體映射檔案的話,可以先本地測驗一下性能,如果是SSD的話性能還是很不錯的,綜合跑下來跟直接寫入記憶體速度相差不會超過一兩倍(記憶體使用率較高的話會嚴重降低性能,甚至會OOM),而且這其中還有很多的優化空間
今天的文章只是提了個思路,細節還有很多要考慮,有疑問的話歡迎提問交流~~
轉載請註明出處,本文鏈接:https://www.uj5u.com/qita/67552.html
標籤:其他
