TensorFlow|創建影像資料集，按檔案名標記-有解無憂

我正在嘗試創建 TensorFlow 資料集來訓練我的模型。我有一個充滿標記照片的檔案夾，標記是檔案名的一部分。

您是否有一種合理的方法來加載資料集進行訓練而不將其拆分到不同的目錄？

示例：對于檔案：

./dataset/path/img0_cat.bmp
./dataset/path/img1_dog.bmp
./dataset/path/img2_horse.bmp
./dataset/path/img3_cat.bmp
./dataset/path/img4_dog.bmp
./dataset/path/img5_horse.bmp
./dataset/path/img6_dog.bmp
./dataset/path/img7_cat.bmp
./dataset/path/img8_horse.bmp
./dataset/path/img9_cat.bmp
./dataset/path/img10_dog.bmp

預期輸出：tf.Dataset 標記為（貓、狗、馬）的熱門

uj5u.com熱心網友回復：

您可以嘗試為每個路徑分配一個 ID，并根據您在訓練集上使用的任何 ID 收集路徑。

如果您使用的是 Tensorflow，Dataset檔案提供了加載資料的資訊方法。具體來說，

dataset_dog = tf.data.Dataset.list_files("./dataset/path/*dog.bmp)

uj5u.com熱心網友回復：

這是整個辣醬玉米餅餡。我更喜歡將 pandas 資料集與 ImageDataGenerator.flow_from_dataframe 一起使用，因為它很靈活。我創建了一個單獨的目錄，其中包含 10 張起重機影像和 10 張信天翁影像。檔案名的格式為 0_crane.jpf、1_crane.jpg 等 ... 10_albatross、11_albatross ..... 下面的代碼處理此目錄。創建一個資料框 df，然后將其拆分為一個 train_df、valid_df 和一個 test_df。然后為 train_gen、test_gen 和 valid_gen 創建 3 個影像資料生成器。我使用了我通常使用的標準模型并訓練了模型。然后以 100% 的準確率評估測驗步驟。代碼如下

import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Dense, Activation,Dropout,Conv2D, MaxPooling2D,BatchNormalization
from tensorflow.keras.optimizers import Adam, Adamax
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras import regularizers
from tensorflow.keras.models import Model

sdir = r'c:\temp\single'
filepaths=[]
labels=[]
flist=os.listdir(sdir)
for f in flist:
    fpath=os.path.join(sdir,f)   
    filepaths.append(fpath)
    index1=f.rfind('_') 1
    index2=f.rfind('.')    
    klass=f[index1:index2]     
    labels.append(klass)
Fseries=pd.Series(filepaths, name='filepaths')
Lseries=pd.Series(labels, name='labels')
df=pd.concat([Fseries, Lseries], axis=1)
# now you can use train_test_split to create a train_df, a test_df and a valid_df
trsplit=.8
vsplit=.1
dsplit=vsplit/(1-trsplit)
train_df, dummy_df=train_test_split (df, train_size=trsplit, shuffle=True, random_state=123, stratify=df['labels'])
valid_df, test_df= train_test_split(dummy_df, train_size=dsplit, shuffle=True, random_state=123, stratify =dummy_df['labels'])
print ('train_df length: ', len(train_df), ' test_df length: ', len(test_df) ,'  valid_df length: ', len(valid_df))

# create a train_gen,  a valid_gen and a test_gen
# for trgen you can specify augmentations like horizontal_flip, vertical_flip etc
img_size=(224,224)
batch_size=16
trgen=ImageDataGenerator(horizontal_flip=True,rotation_range=20, width_shift_range=.2,
                                  height_shift_range=.2, zoom_range=.2   )
tvgen=ImageDataGenerator()
train_gen=trgen.flow_from_dataframe(train_df, x_col='filepaths', y_col='labels',target_size=img_size,
                                   class_mode='categorical', color_mode='rgb', shuffle=True, batch_size=batch_size)
valid_gen=tvgen.flow_from_dataframe(valid_df, x_col='filepaths', y_col='labels',target_size=img_size,
                                   class_mode='categorical', color_mode='rgb', shuffle=False, batch_size=2)
test_gen=tvgen.flow_from_dataframe(test_df, x_col='filepaths', y_col='labels',target_size=img_size,
                                   class_mode='categorical', color_mode='rgb', shuffle=False, batch_size=2)
train_files=train_gen.filenames
classes=list(train_gen.class_indices.keys())
class_count=len(classes)
labels=train_gen.labels
images, labels=next(train_gen)
plt.figure(figsize=(20, 12))
for i in range (len(images)):
    plt.subplot(4,4, i 1) 
    index=np.argmax(labels[i])
    class_name=classes[index]
    plt.title(class_name, color='yellow', fontsize=18)   
    plt.axis('off')
    plt.imshow(images[i]/255)
plt.show()
# build a model
img_shape=(img_size[0], img_size[1], 3)
model_name='EfficientNetB5'
base_model=tf.keras.applications.efficientnet.EfficientNetB5(include_top=False, weights="imagenet",input_shape=img_shape, pooling='max') 
# Note you are always told NOT to make the base model trainable initially- that is WRONG you get better results leaving it trainable
base_model.trainable=True
x=base_model.output
x=BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
x = Dense(1024, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
                bias_regularizer=regularizers.l1(0.006) ,activation='relu')(x)
x=Dropout(rate=.3, seed=123)(x)
x = Dense(128, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
                bias_regularizer=regularizers.l1(0.006) ,activation='relu')(x)
x=Dropout(rate=.45, seed=123)(x)        
output=Dense(class_count, activation='softmax')(x)
model=Model(inputs=base_model.input, outputs=output)
lr=.001 # start with this learning rate
model.compile(Adamax(learning_rate=lr), loss='categorical_crossentropy', metrics=['accuracy']) 

epochs=5
history=model.fit(x=train_gen,  epochs=epochs, verbose=1,  validation_data=valid_gen,
               validation_steps=None,  shuffle=False,  initial_epoch=0)

loss, acc =model.evaluate(test_gen)
print (' accuracy on test set is: ', acc* 100, '%')

uj5u.com熱心網友回復：

謝謝大家的回復

我解決的方法如下：

classNames = ['dog', 'cat', 'horse']

def getLabel(file_path):
    # Convert the path to a list of path components
    fileName = tf.strings.split(file_path, os.path.sep)[-1]
    # get label name from filename
    className = tf.strings.split(fileName, '_')[1]
    className = tf.strings.split(className, '.')[0]
    # get one_hot vector boolean
    one_hot = className == classNames
    # cast vector type to integer 
    return tf.cast(one_hot, dtype=tf.int8, name=None)

def getImage(file_path):
    # Load the raw data from the file as a string
    img = tf.io.read_file(file_path)
    # Convert the compressed string to a 3D uint8 tensor
    img = tf.io.decode_bmp(img, channels=3)
    # cast tf.Tensor type to uint8 
    return tf.cast(img, dtype=tf.uint8, name=None)

def process_path(file_path):
    label = getLabel(file_path)
    img = getImage(file_path)
    return img, label

path = './dataset/path/*.bmp'
ds = tf.data.Dataset.list_files(path)
ds = ds.map(process_path)

在此程序結束時，您將獲得一個 Tensorflow 可訓練資料集（批量大小需要更多配置，請參閱參考資料），標簽為 one-hot 向量。

運行時：

for image, label in ds.take(5):
     imageShape = image.numpy().shape
     label = label.numpy()
     labelName = class_names[np.argmax(label)]
     print('Image Shape: {}, Label: {}, LabelName: {}'.format(imageShape, label, labelName))

你得到了：

Image Shape: (180, 180, 3), Label: [1 0 0], LabelName: dog
Image Shape: (180, 180, 3), Label: [0 1 0], LabelName: cat
Image Shape: (180, 180, 3), Label: [0 0 1], LabelName: horse
Image Shape: (180, 180, 3), Label: [1 0 0], LabelName: dog
Image Shape: (180, 180, 3), Label: [0 1 0], LabelName: cat

參考： https ://www.tensorflow.org/tutorials/load_data/images

轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/456248.html

標籤：Python 张量流深度学习 tf.dataset

上一篇：Keras早期停止和監控

下一篇：CNN準確度繪圖