我有一個大資料，我想將其加載到 Tensorflow 資料集中以訓練 LSTM 網路。由于資料的大小，我想使用流功能而不是將整個資料讀入記憶體。我正在努力閱讀我的資料，以便每個樣本i都正確地成形為(t i , m)。
要復制的示例代碼：
# One hundred samples, each with three features
# Second dim is time-steps for each sample. I will
# randomize this in a step below
x = np.random.randn(100,10,3)
# One hundred {0,1} labels
y = (np.random.rand(100)>0.5)*1
y=y.reshape((-1,1))
# Save each sample in its own file
for i in range(len(x)):
cat = y[i][0]
data = x[i]
# Simulate random length of each sample
data = data[:np.random.randint(4,10),:]
fname = 'tmp_csv/{:.0f}/{:03.0f}.csv'.format(cat,i)
np.savetxt(fname, data, delimiter=',')
現在我有一百個 csv 檔案，每個檔案都有一個大小為(t i , 3) 的樣本。如何在保持每個樣本的形狀的同時將這些檔案讀回 Tensorflow 資料集？
我嘗試了序列化（但不知道如何正確執行），展平使每個樣本都在一行中（但不知道如何處理可變行大小以及如何重塑），并且我嘗試了 vanilla make_csv_dataset。這是我的 make_csv_dataset 嘗試：
ds = tf.data.experimental.make_csv_dataset(
file_pattern = "tmp_csv/*/*.csv",
batch_size=10, num_epochs=1,
num_parallel_reads=5,
shuffle_buffer_size=10,
header=False,
column_names=['a','b','c']
)
for i in ds.take(1):
print(i)
...但這導致每個樣本的形狀為 (1,3)。
uj5u.com熱心網友回復：
問題是將make_csv_dataset每個csv檔案中的每一行都解釋為一個樣本。您可以嘗試這樣的操作，但我不確定它對您的用例的效率如何：
import tensorflow as tf
import numpy as np
# One hundred samples, each with three features
# Second dim is time-steps for each sample. I will
# randomize this in a step below
x = np.random.randn(100,10,3)
# One hundred {0,1} labels
y = (np.random.rand(100)>0.5)*1
y=y.reshape((-1,1))
# Save each sample in its own file
for i in range(len(x)):
cat = y[i][0]
data = x[i]
# Simulate random length of each sample
data = data[:np.random.randint(4,10),:]
fname = 'tmp_csv/{:.0f}{:03.0f}.csv'.format(cat,i)
np.savetxt(fname, data, delimiter=',')
def columns_to_tensor(data_from_one_csv):
ta = tf.TensorArray(dtype=tf.float32, size=0, dynamic_size=True)
for i, t in enumerate(data_from_one_csv):
ta = ta.write(tf.cast(i, dtype=tf.int32), tf.stack([t[0], t[1], t[2]], axis=0))
return ta.stack()
files = tf.data.Dataset.list_files("tmp_csv/*.csv")
ds = files.map(lambda file: tf.data.experimental.CsvDataset(file, record_defaults=[tf.float32, tf.float32, tf.float32], header=False))
ds = ds.map(columns_to_tensor)
for i,j in enumerate(ds):
print(i, j.shape)
0 (5, 3)
1 (9, 3)
2 (5, 3)
3 (6, 3)
4 (8, 3)
5 (7, 3)
6 (6, 3)
7 (8, 3)
8 (8, 3)
9 (7, 3)
10 (9, 3)
11 (9, 3)
12 (7, 3)
13 (9, 3)
14 (4, 3)
15 (5, 3)
16 (6, 3)
17 (6, 3)
18 (8, 3)
19 (8, 3)
20 (8, 3)
21 (9, 3)
22 (9, 3)
23 (7, 3)
24 (8, 3)
25 (8, 3)
26 (5, 3)
27 (7, 3)
28 (5, 3)
29 (8, 3)
30 (9, 3)
31 (6, 3)
32 (6, 3)
33 (7, 3)
34 (6, 3)
35 (9, 3)
36 (9, 3)
37 (5, 3)
38 (9, 3)
39 (9, 3)
40 (7, 3)
41 (7, 3)
42 (7, 3)
43 (6, 3)
44 (9, 3)
45 (4, 3)
46 (9, 3)
47 (6, 3)
48 (9, 3)
49 (8, 3)
50 (7, 3)
51 (4, 3)
52 (4, 3)
53 (6, 3)
54 (7, 3)
55 (7, 3)
56 (9, 3)
57 (7, 3)
58 (5, 3)
59 (7, 3)
60 (8, 3)
61 (8, 3)
62 (5, 3)
63 (5, 3)
64 (7, 3)
65 (6, 3)
66 (6, 3)
67 (7, 3)
68 (6, 3)
69 (9, 3)
70 (5, 3)
71 (4, 3)
72 (8, 3)
73 (8, 3)
74 (6, 3)
75 (7, 3)
76 (9, 3)
77 (6, 3)
78 (5, 3)
79 (7, 3)
80 (6, 3)
81 (5, 3)
82 (4, 3)
83 (5, 3)
84 (4, 3)
85 (5, 3)
86 (4, 3)
87 (4, 3)
88 (7, 3)
89 (5, 3)
90 (4, 3)
91 (7, 3)
92 (4, 3)
93 (7, 3)
94 (4, 3)
95 (5, 3)
96 (6, 3)
97 (6, 3)
98 (7, 3)
99 (9, 3)
之后，只需呼叫ds.batch所需的批量大小即可。
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/375445.html
上一篇：自動編碼器：“model_3”層需要1個輸入，但它收到64個輸入張量
下一篇：關于基于秩的計算的自動微分