我想將 tensorflow2 資料集拆分為兩個資料集，一個包含“特征”，另一個包含“標簽”。
在資料集中，每個元素都是一個 python 字典：
{'id': 29, 'val1': [0.97, 0.52], 'val2': [627], 'val3': ['baseball', 'football', 'basketball'], 'label': 1.0}
{'id': 11, 'val1': [0.22, 0.36], 'val2': [81], 'val3': ['swimming', 'running', 'jumpoing'], 'label': 0.0}
該資料集是一個預取的資料集，從 1000 個 txt (.gz) 檔案中加載。每個檔案的大小為 500KB-600KB。所有檔案資料都無法加載到記憶體中，所以我必須分批預取（大小為 200）。
def a_func(file_paths, batch_size=200):
dataset = tf.data.TFRecordDataset(file_paths, compression_type='GZIP')
dataset = dataset.batch(batch_size)
dataset = dataset.prefetch(batch_size)
#split the dataset into features and labels ?
label = dataset.map(lambda x: col for col in x if col=='label') # error: "x" not defined
features = dataset.map(lambda x: col for col in x if col!='label')
return features, label
如何按列名將資料集拆分為兩個資料集？
更新
ds = tf.data.TFRecordDataset(file_paths, compression_type='GZIP')
type(ds)
# tensorflow.python.data.ops.readers.TFRecordDatasetV2
label = ds.map(lambda x: x['label'])
# TypeError: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices, got 'label'
for row in ds:
print(row.numpy()) # show a lot of hex chars
uj5u.com熱心網友回復：
IIUC，你可以嘗試這樣的事情：
import tensorflow as tf
# Create dummy data
ds1 = tf.data.Dataset.from_tensors(({'id': 11, 'val1': [0.22, 0.36], 'val2': [81], 'val3': ['swimming', 'running', 'jumpoing'], 'label': 0.0}))
ds2 = tf.data.Dataset.from_tensors(({'id': 29, 'val1': [0.97, 0.52], 'val2': [627], 'val3': ['baseball', 'football', 'basketball'], 'label': 1.0}))
ds = ds1.concatenate(ds2)
label = ds.map(lambda x: x['label'])
features = ds.map(lambda x: (x['id'], x['val1'], x['val2'], x['val3']))
for l in label:
print(l)
for f in features:
print(f)
tf.Tensor(0.0, shape=(), dtype=float32)
tf.Tensor(1.0, shape=(), dtype=float32)
(<tf.Tensor: shape=(), dtype=int32, numpy=11>, <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.22, 0.36], dtype=float32)>, <tf.Tensor: shape=(1,), dtype=int32, numpy=array([81], dtype=int32)>, <tf.Tensor: shape=(3,), dtype=string, numpy=array([b'swimming', b'running', b'jumpoing'], dtype=object)>)
(<tf.Tensor: shape=(), dtype=int32, numpy=29>, <tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.97, 0.52], dtype=float32)>, <tf.Tensor: shape=(1,), dtype=int32, numpy=array([627], dtype=int32)>, <tf.Tensor: shape=(3,), dtype=string, numpy=array([b'baseball', b'football', b'basketball'], dtype=object)>)
或使用動態鍵：
import tensorflow as tf
import numpy as np
# Create dummy data
ds1 = tf.data.Dataset.from_tensors(({'id': 11, 'val1': [0.22, 0.36], 'val2': [81], 'val3': ['swimming', 'running', 'jumpoing'], 'label': 0.0}))
ds2 = tf.data.Dataset.from_tensors(({'id': 29, 'val1': [0.97, 0.52], 'val2': [627], 'val3': ['baseball', 'football', 'basketball'], 'label': 1.0}))
ds = ds1.concatenate(ds2)
label = ds.map(lambda x: x['label'])
features = ds.map(lambda x: (list(map(x.get, list(np.setdiff1d(list(x.keys()),['label']))))))
轉載請註明出處，本文鏈接：https://www.uj5u.com/qukuanlian/432302.html
上一篇：獲取`ValueError：`logits`和`labels`必須具有相同的形狀，收到（（None，1）vs（））`錯誤