今天早些時候這對我來說作業得很好，但是當我重新啟動我的筆記本時它突然開始表現得很奇怪。我有一個 tf 資料集，它接受 numpy 檔案及其相應的標簽作為輸入，就像這樣tf.data.Dataset.from_tensor_slices((specgram_files, labels))。當我使用 1 項時，for item in ds.take(1): print(item)我得到了預期的輸出，這是一個張量元組，其中第一個張量包含 numpy 檔案的名稱作為位元組字串，第二個張量包含編碼標簽。然后我有一個函式，它使用讀取檔案np.load()并生成一個 numpy 陣列，然后回傳該陣列。這個函式被傳遞給 map() 方法，它看起來像這樣：
ds = ds.map(
lambda file, label: tuple([tf.numpy_function(read_npy_file, [file], [tf.float32]), label]),
num_parallel_calls=tf.data.AUTOTUNE)
，其中 read_npy_file 看起來像這樣：
def read_npy_file(data):
# 'data' stores the file name of the numpy binary file storing the features of a particular sound file
# as a bytes string.
# decode() is called on the bytes string to decode it from a bytes string to a regular string
# so that it can passed as a parameter into np.load()
data = np.load(data.decode())
return data.astype(np.float32)
如您所見，映射應該創建另一個張量元組，其中第一個張量是 numpy 陣列，第二個張量是標簽，未觸及。這在早期作業得很好，但現在它給出了最奇怪的行為。我在函式中放置了列印陳述句，read_npy_file()以查看是否傳入了正確的資料。我希望它傳遞一個位元組字串，但是當我呼叫函式并從資料集中獲取 1 個專案來觸發一個時print(data)，它會產生這個輸出read_npy_file()映射使用ds.take(1)：
b'./challengeA_data/log_spectrogram/2603ebb3-3cd3-43cc-98ef-0c128c515863.npy'b'./challengeA_data/log_spectrogram/fab6a266-e97a-4935-a0c3-444fc4426fc5.npy'b'./challengeA_data/log_spectrogram/93014682-60a2-45bd-9c9e-7f3c97b83be9.npy'b'./challengeA_data/log_spectrogram/710f2430-5da3-4822-a252-6ad3601b92d9.npy'b'./challengeA_data/log_spectrogram/e757058c-91de-4381-8184-65f001c95647.npy'
b'./challengeA_data/log_spectrogram/38b12689-04ba-422b-a972-5856b05ca868.npy'
b'./challengeA_data/log_spectrogram/7c9ccc04-a2d2-4eec-bafd-0c97b3658c26.npy'b'./challengeA_data/log_spectrogram/c7cc3520-7218-4d07-9f0a-6bd7bb90a551.npy'
b'./challengeA_data/log_spectrogram/21f6060a-9766-4810-bd7c-0437f47ccb98.npy'
我沒有修改輸出的任何格式。
我將不勝感激任何幫助。與 TFDS 合作絕對是一場噩夢，哈哈。
這是完整的代碼
def read_npy_file(data):
# 'data' stores the file name of the numpy binary file storing the features of a particular sound file
# as a bytes string.
# decode() is called on the bytes string to decode it from a bytes string to a regular string
# so that it can passed as a parameter into np.load()
print(data)
data = np.load(data.decode())
return data.astype(np.float32)
specgram_ds = tf.data.Dataset.from_tensor_slices((specgram_files, labels))
specgram_ds = specgram_ds.map(
lambda file, label: tuple([tf.numpy_function(read_npy_file, [file], [tf.float32]), label]),
num_parallel_calls=tf.data.AUTOTUNE)
num_files = len(train_df)
num_train = int(0.8 * num_files)
num_val = int(0.1 * num_files)
num_test = int(0.1 * num_files)
specgram_ds = specgram_ds.shuffle(buffer_size=1000)
specgram_train_ds = specgram_ds.take(num_train)
specgram_test_ds = specgram_ds.skip(num_train)
specgram_val_ds = specgram_test_ds.take(num_val)
specgram_test_ds = specgram_test_ds.skip(num_val)
# iterating over one item to trigger the mapping function
for item in specgram_ds.take(1):
pass
謝謝！
uj5u.com熱心網友回復：
你的邏輯似乎很好。您實際上只是在觀察與tf.data.AUTOTUNE結合使用的行為print(*)。根據檔案：
如果使用值 tf.data.AUTOTUNE，則并行呼叫的數量是根據可用 CPU 動態設定的。
您可以運行以下代碼幾次來觀察變化：
import tensorflow as tf
import numpy as np
def read_npy_file(data):
# 'data' stores the file name of the numpy binary file storing the features of a particular sound file
# as a bytes string.
# decode() is called on the bytes string to decode it from a bytes string to a regular string
# so that it can passed as a parameter into np.load()
print(data)
data = np.load(data.decode())
return data.astype(np.float32)
# Create dummy data
for i in range(4):
np.save('{}-array'.format(i), np.random.random((5,5)))
specgram_files = ['/content/0-array.npy', '/content/1-array.npy', '/content/2-array.npy', '/content/3-array.npy']
labels = [1, 0, 0, 1]
specgram_ds = tf.data.Dataset.from_tensor_slices((specgram_files, labels))
specgram_ds = specgram_ds.map(
lambda file, label: tuple([tf.numpy_function(read_npy_file, [file], [tf.float32]), label]),
num_parallel_calls=tf.data.AUTOTUNE)
num_files = len(specgram_files)
num_train = int(0.8 * num_files)
num_val = int(0.1 * num_files)
num_test = int(0.1 * num_files)
specgram_ds = specgram_ds.shuffle(buffer_size=1000)
specgram_train_ds = specgram_ds.take(num_train)
specgram_test_ds = specgram_ds.skip(num_train)
specgram_val_ds = specgram_test_ds.take(num_val)
specgram_test_ds = specgram_test_ds.skip(num_val)
for item in specgram_ds.take(1):
pass
也看到這個。最后，請注意，使用tf.print而不是print應該消除任何副作用。
轉載請註明出處，本文鏈接：https://www.uj5u.com/caozuo/489711.html