訓練FF神經語言模型

考慮句子“The cat is upstairs”的 3-gram，其中每個單詞都用 @ 和 ~ 符號分隔。

trigrams = ['@th', 'the', 'he~', '@ca', 'cat', 'at~', '@is', 'is~', 
             '@up', 'ups', 'pst', 'sta', 'tai', 'air', 'irs', 'rs~']

我想用這句話訓練一個基于字符的前饋神經語言模型，但我無法正確擬合 X 和 y 引數。

我的代碼如下：

# trigrams encoded
d = dict([(y,x 1) for x,y in enumerate(sorted(set(trigrams)))])
trigrams_encoded = [d[x] for x in trigrams]
# trigrams_encoded = [3, 15, 8, 1, 7, 6, 2, 10, 4, 16, 11, 13, 14, 5, 9, 12]

# x_train
x_train = [] # list of lists, each list contains 3 encoded trigrams
for i in range(len(trigrams_encoded)-3) :
    lst = trigrams_encoded[i:i 3]
    x_train.append(lst)
x_train = np.array(x_train)            # x_train shape is (13,3)

# y_train
y_train = trigrams_encoded[3:]
data = np.array(y_train)
y_onehot = to_categorical(data)        # y_onehot shape is (13,17)
y_onehot = np.delete(y_onehot, 0, 1)   # now shape is (13,16)

# define model
model = Sequential()
model.add(Embedding(len(d), 10, input_length=3)) #len(d) = 16
model.add(Flatten())
model.add(Dense(10, activation='relu'))
model.add(Dense(len(d), activation='softmax'))

# compile the model
# i have set sparse_categorical_crossentropy here, but not sure if this is correct. feel free to change it
model.compile(loss="sparse_categorical_crossentropy", optimizer='adam', metrics=['accuracy'])

# train the model
model.fit(x_train, y_onehot, epochs=1, verbose=0)

我最初的嘗試是說，由于 input_length=3，模型將作為列出的 n-gram 的輸入三元組，這些 n-gram 應該被標記為串列中的下一個 n-gram。但這似乎失敗了。（它應該失敗嗎？）

上面的代碼引發了以下我不知道如何解決的錯誤：

"InvalidArgumentError: Graph execution error:

Detected at node 'sequential/embedding/embedding_lookup' defined at (most recent call last):

(... many lines...)

Node: 'sequential/embedding/embedding_lookup'
indices[5,1] = 16 is not in [0, 16)"

您能否在此處協助正確選擇 X 和 y ？

uj5u.com熱心網友回復：

您的代碼在用作損失函式時運行良好categorical_crossentropy，因為您使用的是單熱編碼標簽：

import numpy as np
import tensorflow as tf

trigrams = ['@th', 'the', 'he~', '@ca', 'cat', 'at~', '@is', 'is~', 
             '@up', 'ups', 'pst', 'sta', 'tai', 'air', 'irs', 'rs~']


# trigrams encoded
d = dict([(y,x 1) for x,y in enumerate(sorted(set(trigrams)))])
trigrams_encoded = [d[x] for x in trigrams]
# trigrams_encoded = [3, 15, 8, 1, 7, 6, 2, 10, 4, 16, 11, 13, 14, 5, 9, 12]

# x_train
x_train = [] # list of lists, each list contains 3 encoded trigrams
for i in range(len(trigrams_encoded)-3) :
    lst = trigrams_encoded[i:i 3]
    x_train.append(lst)
x_train = np.array(x_train)            # x_train shape is (13,3)

# y_train
y_train = trigrams_encoded[3:]
data = np.array(y_train)
y_onehot = tf.keras.utils.to_categorical(data)        # y_onehot shape is (13,17)
y_onehot = np.delete(y_onehot, 0, 1)   # now shape is (13,16)

# define model
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(len(d)   1, 10, input_length=3)) #len(d) = 16
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(10, activation='relu'))
model.add(tf.keras.layers.Dense(len(d), activation='softmax'))

model.compile(loss="categorical_crossentropy", optimizer='adam', metrics=['accuracy'])

# train the model
model.fit(x_train, y_onehot, epochs=5, verbose=1)

sparse_categorical_crossentropy僅適用于稀疏整數值。

標籤：Python 张量流 喀拉斯 神经网络 语言模型

上一篇：將h5轉換為tflite

下一篇：多輸出損失keras/tensorflow的表現不如預期。目標#2MAE損失巨大且>1，盡管sigmoid激活和0-1縮放標簽