我有的：
- 一個神經網路模型
- 10 個結構相同的資料集
我想要的是：
- 分別在所有資料集上訓練模型
- 分別保存他們的模型
我可以單獨訓練資料集并一次保存一個模型。但是我想在一次運行中加載我的 10 個資料集并使用它們創建 10 個模型。解決方案可能很明顯，但我對此很陌生。我如何實作這一目標？
提前致謝。
uj5u.com熱心網友回復：
您可以使用 的概念之一concurrency and parallelism，即Multi-Threading，或在某些情況下，Multi-Processing來實作此目的。
最簡單的編碼方法是使用concurrent-futurespython 模塊。
您可以為每個要使用的資料集呼叫模型上的訓練函式，所有這些都在 ThreadPoolExecutor 下，以便觸發并行執行緒來執行單獨的訓練。
代碼可能有點像這樣：
第 1 步：必要的進口
from concurrent.futures import ThreadPoolExecutor, as_completed
import tensorflow as tf
from tensorflow.keras.models import load_model, Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
第 2 步：創建和構建模型
def create_model(): # responsible for creating model
model = Sequential()
model.add(Flatten()) # adding NN layers
model.add(Dense(64))
model.add(Activation('relu'))
# ........ so on
model.compile(optimizer='..', loss='..', metrics=[...]) # compiling the model
return model # finally returning the model
第三步：定義擬合函式（執行模型訓練）
def fit(model, XY_train): # performs model.fit(...parameters...)
model.fit(XY_train[0], XY_train[1], epochs=5, validation_split=0.3) # use your already defined x_train, y_train
return model # finally returns trained model
第 4 步：并行訓練器方法，使用 TPE 背景關系管理器觸發同步訓練
# trains provided model on each dataset parallelly by using multi-threading
def parallel_trainer(model, XY_train_datasets : list[tuple]):
with ThreadPoolExecutor(max_workers = len(XY_train_datasets)) as executor:
futureObjs = [
executor.submit(
lambda ds: fit(model, ds), XY_train_datasets) # Call Fit for each dataset iterate through the datasets
]
for i, obj in enumerate(as_completed(futureObjs)): # iterate through trained models
(obj.result()).save(f"{i}.model") # save models
第 5 步：創建模型、加載資料集、呼叫并行訓練器
model = create_model() # create the model
mnist = tf.keras.datasets.mnist # get dataset - for example :- mnist dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data() # get (x_train, y_train), (x_test, y_test)
datasets = [(x_train, y_train)]*10 # list of dataset paths (in your case, same dataset used 10 times)
parallel_trainer(model, datasets) # call parallel trainer
整個節目
from concurrent.futures import ThreadPoolExecutor, as_completed
import tensorflow as tf
from tensorflow.keras.models import load_model, Sequential
from tensorflow.keras.layers import Dense, Activation, Flatten
def create_model(): # responsible for creating model
model = Sequential()
model.add(Flatten()) # adding NN layers
model.add(Dense(64))
model.add(Activation('relu'))
# ........ so on
model.compile(optimizer='..', loss='..', metrics=[...]) # compiling the model
return model # finally returning the model
def fit(model, XY_train): # performs model.fit(...parameters...)
model.fit(XY_train[0], XY_train[1], epochs=5, validation_split=0.3) # use your already defined x_train, y_train
return model # finally returns trained model
# trains provided model on each dataset parallelly by using multi-threading
def parallel_trainer(model, XY_train_datasets : list[tuple]):
with ThreadPoolExecutor(max_workers = len(XY_train_datasets)) as executor:
futureObjs = [
executor.submit(
lambda ds: fit(model, ds), XY_train_datasets) # Call Fit for each dataset iterate through the datasets
]
for i, obj in enumerate(as_completed(futureObjs)): # iterate through trained models
(obj.result()).save(f"{i}.model") # save models
model = create_model() # create the model
mnist = tf.keras.datasets.mnist # get dataset - for example :- mnist dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data() # get (x_train, y_train), (x_test, y_test)
datasets = [(x_train, y_train)]*10 # list of dataset paths (in your case, same dataset used 10 times)
parallel_trainer(model, datasets) # call parallel trainer
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/401993.html