我有一個簡單的例子，一個有 8 列名為 feature_# 的鑲木地板檔案，每列填充 1 到 100
feature_1 feature_2 ... feature_8
1 1 1
2 2 2
... ... ...
99 99 99
100 100 100
我的模型：
all_cols = ["feature_1","feature_2","feature_3","feature_4","feature_5","feature_6","feature_7","feature_8"]
x_cols = ["feature_1","feature_2","feature_3","feature_4","feature_5","feature_6","feature_7"]
inputs = [Input(shape=(1,),name=col) for col in x_cols]
merged = Concatenate(axis=1)(inputs)
x = Dense(50, activation="relu")(merged)
x = Dense(20,activation="relu")(x)
outputs = Dense(101,activation="softmax")(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
opt = tf.keras.optimizers.Adam(learning_rate=.001)
model.compile(loss="sparse_categorical_crossentropy",\
optimizer=opt,metrics=['accuracy'])
我像這樣使用 petastorm：
batch_size = 4
with make_batch_reader('%s/df_100.parquet' % data_dir, num_epochs=1,
schema_fields=all_cols) as train_reader:
with make_batch_reader('%s/df_100.parquet' % data_dir, num_epochs=1,
schema_fields=all_cols) as val_reader:
train_ds = make_petastorm_dataset(train_reader) \
.unbatch() \
.map(
lambda x: (tuple(getattr(x, col) for col in x_cols),getattr(x,"feature_8"))
) \
.batch(batch_size)
val_ds = make_petastorm_dataset(val_reader) \
.unbatch() \
.map(
lambda x: (tuple(getattr(x, col) for col in x_cols),
getattr(x,"feature_8"))
) \
.batch(batch_size)
對于這個簡單的示例，我將相同的資料用于訓練作為驗證。我想確認整個資料集都進入了 model.fit() 所以我寫了一個自定義回呼
class MyCustomCallback(tf.keras.callbacks.Callback):
def __init__(self, train_data):
self.mylist = []
self.train_data = train_data
def on_train_batch_begin(self, batch, logs=None):
print(list(self.train_data.take(1).as_numpy_iterator())[0][0][0])
# and I pass the dataset to the custom callback:
callbacks.append(MyCustomCallback(train_ds))
不會列印所有值... 1 到 100。如果我在沒有 model.fit 的情況下迭代資料集（簡單的 for 回圈），那么我確實得到了 1 到 100，所以我認為 take() 正在與model.fit，只是一個理論。
我也試過：
class MyCustomCallback(tf.keras.callbacks.Callback):
def on_train_batch_begin(self, batch, logs=None):
print(self.model.layers[0].input) # or .output
#or
#print(self.model.layers[0].get_weights())
但這并沒有給我任何真正的價值，并且 get_weights() 列印出空陣列
這是列印輸入列印出來的內容：
KerasTensor(type_spec=TensorSpec(shape=(None, 1), dtype=tf.float32, name='feature_1'), name='feature_1', description="created by layer 'feature_1'")
我也嘗試在圖層的輸入和輸出上使用 K.eval() ，結果是一個 numpy 問題，任何急切的設定都不能解決這個問題。
我真的不認為這應該這么難。我只想在資料集進入訓練之前達到峰值。
我玩弄了 repeat()、cache()，并在 model.fit 之前簡單地迭代資料集，但我不喜歡這種情況發生在 model.fit 之前，除非它被快取，否則它會重新洗牌，等等...
但我也希望能夠在任何時候任意查看模型，任何值，任何重量。我不覺得我可以訪問這些東西，但覺得我應該可以。
任何幫助表示贊賞。
哦，使用 tensorflow 2.6.2 atm 和 tf.keras
uj5u.com熱心網友回復：
我認為這完全取決于您的大小，batch_size因為take(1)需要一批，如果batch_size< 100，您將看不到所有值。例如，如果您有batch_size=100，那么您肯定會看到 1 到 100 的值：
import pandas as pd
import tensorflow as tf
import numpy as np
from petastorm.tf_utils import make_petastorm_dataset
from petastorm.reader import make_batch_reader
df = pd.DataFrame({'feature1':np.arange(1, 101),
'feature2':np.arange(1, 101),
'feature3':np.arange(1, 101),
'feature4':np.arange(1, 101),
'feature5':np.arange(1, 101),
'feature6':np.arange(1, 101),
'feature7':np.arange(1, 101),
'feature8':np.arange(1, 101)})
columns = list(df)
df.to_parquet('file.parquet')
x_cols = columns[:-1]
batch_size = 100
class MyCustomCallback(tf.keras.callbacks.Callback):
def __init__(self, train_data):
self.mylist = []
self.train_data = train_data
def on_train_batch_begin(self, batch, logs=None):
tf.print(list(self.train_data.take(1).as_numpy_iterator())[0][0][0])
with make_batch_reader('file:///content/file.parquet', num_epochs=1,
schema_fields=columns) as train_reader:
train_ds = make_petastorm_dataset(train_reader) \
.unbatch() \
.map(
lambda x: (tuple(getattr(x, col) for col in x_cols),getattr(x,"feature8"))
) \
.shuffle(buffer_size=1000).batch(batch_size)
inputs = [tf.keras.layers.Input(shape=(1,),name=col) for col in x_cols]
merged = tf.keras.layers.Concatenate(axis=1)(inputs)
x = tf.keras.layers.Dense(50, activation="relu")(merged)
x = tf.keras.layers.Dense(20,activation="relu")(x)
outputs = tf.keras.layers.Dense(101, activation="softmax")(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
opt = tf.keras.optimizers.Adam(learning_rate=.001)
model.compile(loss="sparse_categorical_crossentropy", optimizer=opt,metrics=['accuracy'])
model.fit(train_ds, epochs=2, callbacks=[MyCustomCallback(train_ds)])
Epoch 1/2
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65,
66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78,
79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
92, 93, 94, 95, 96, 97, 98, 99, 100])
1/Unknown - 1s 777ms/step - loss: 19.3339 - accuracy: 0.0100array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65,
66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78,
79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91,
92, 93, 94, 95, 96, 97, 98, 99, 100])
1/1 [==============================] - 1s 899ms/step - loss: 19.3339 - accuracy: 0.0100
...
另外，我不確定究竟有什么好處petastorm，但如果您正在尋找替代方案，您可以嘗試tensorflow-io：
import pandas as pd
import tensorflow_io as tfio
import tensorflow as tf
import numpy as np
df = pd.DataFrame({'feature1':np.arange(1, 101),
'feature2':np.arange(1, 101),
'feature3':np.arange(1, 101),
'feature4':np.arange(1, 101),
'feature5':np.arange(1, 101),
'feature6':np.arange(1, 101),
'feature7':np.arange(1, 101),
'feature8':np.arange(1, 101)})
columns = list(df)
df.to_parquet('file.parquet')
ds = tfio.IODataset.from_parquet('file.parquet', columns = columns)
x_cols = columns[:-1]
batch_size = 100
train_ds = ds.map(lambda x: (tuple(x[col] for col in x_cols),x["feature8"])).shuffle(buffer_size=1000).batch(batch_size)
inputs = [tf.keras.layers.Input(shape=(1,),name=col) for col in x_cols]
merged = tf.keras.layers.Concatenate(axis=1)(inputs)
x = tf.keras.layers.Dense(50, activation="relu")(merged)
x = tf.keras.layers.Dense(20,activation="relu")(x)
outputs = tf.keras.layers.Dense(101, activation="softmax")(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)
opt = tf.keras.optimizers.Adam(learning_rate=.001)
model.compile(loss="sparse_categorical_crossentropy", optimizer=opt,metrics=['accuracy'])
model.fit(train_ds, epochs=2, callbacks=[MyCustomCallback(train_ds)])
更新 1：您可以將每個批次添加到Callback每個 epoch 結束時的陣列中，您可以列印值并為下一個 epoch 重置陣列：
class MyCustomCallback(tf.keras.callbacks.Callback):
def __init__(self, train_data):
self.mylist = tf.TensorArray(dtype=tf.int32, size=0, dynamic_size=True, infer_shape=True)
self.train_data = train_data
def on_batch_end(self, batch, logs=None):
self.mylist = self.mylist.write(self.mylist.size(), list(self.train_data.take(1).as_numpy_iterator())[0][0][0])
def on_epoch_end(self, epoch, logs=None):
arr = self.mylist.stack()
tf.print(arr, summarize=-1)
self.mylist = tf.TensorArray(dtype=tf.int32, size=0, dynamic_size=True, infer_shape=True)
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/414250.html
標籤：