幾個小時以來，我一直試圖弄清楚這一點。使下面的程式作業的簡單解決方案是僅使用 keras.MSE，但我想了解為什么我的版本無法正常作業，而不是我希望該程式能夠作業。
在我看來，差值平方的平均值與 keras.MSE 非常接近。我期待差異，但我的似乎開始接近并且變得越來越糟，我不知道為什么。
step=0 theirs= 13.1761 mine= 14.0251
step=5 theirs= 10.3337 mine= 11.8363
…
step=90 theirs= 0.0361 mine= 6.9888
step=95 theirs= 0.0332 mine= 6.9604
我一直在研究 keras 和 tensorflow。我回到了 keras/losses.py backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1)，這看起來和我很相似。雖然我承認 tf.math.squared_difference 中的代碼對我來說意義不大，但它的作業原理與tf.square(y_true-y_pred)ipython 中的大致相同。
我肯定錯過了一些東西。
這是我的小程式：
import tensorflow as tf
import numpy as np
def small_ds():
in_t = tf.cast(np.random.randint(5, size=(24, 2)), tf.float32)
out_t = tf.reduce_sum(in_t, axis=-1)
return in_t, out_t
def small_model():
i = tf.keras.layers.Input(shape=(2,))
d = i
d = tf.keras.layers.Dense(32, activation="LeakyReLU")(d)
d = tf.keras.layers.Dense(32, activation="LeakyReLU")(d)
d = tf.keras.layers.Dense(32, activation="LeakyReLU")(d)
o = tf.keras.layers.Dense(1, activation="LeakyReLU")(d)
m = tf.keras.Model(inputs=i, outputs=o)
return m
def what_is_happening_here():
opt = tf.keras.optimizers.Adam()
tf_mse = tf.keras.losses.MeanSquaredError()
@tf.function
def my_mse(y_true, y_pred):
return tf.reduce_mean(tf.square(y_true-y_pred))
m = small_model()
@tf.function
def train_step(x_input, y_true):
with tf.GradientTape() as tape:
y_pred = m(x_input, training=True)
theirs = tf_mse(y_true, y_pred)
mine = my_mse(y_true, y_pred)
grad = tape.gradient(theirs, m.trainable_variables)
opt.apply_gradients(zip(grad, m.trainable_variables))
return theirs, mine
x_input, y_true = small_ds()
for step in range(100):
theirs, mine = train_step(x_input, y_true)
if (step % 5) == 0:
print(f'step={step} theirs={theirs:8.4f} mine={mine:8.4f}')
if __name__ == '__main__':
what_is_happening_here()
編輯：
I was initially convinced by the first answer, but I don't think it's quite right. If I generate some completely random vectors and run through mine vs theirs with and without reductions, everything is identical. This version of my_mse is slightly difference than the above, but the strange non-linear differences show up as before if I run the modified version through the training loop above.
I think the optimizer or the graph is doing something else that I can't see.
I realize this is a trivial problem in the grand scheme of things, but I'd like to be able to write my own loss functions at some point, and I really don't trust them to work the same as the native ones.
I've also tried wrapping my loss functions in the tf.keras.losses.Loss class, but everything turns out the same (ie, it doesn't work).
In [22]: tf_mse = tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)
...: my_mse = lambda x,y: tf.reduce_mean(tf.square(x-y), axis=-1)
...:
...: tf_mser = tf.keras.losses.MeanSquaredError()
...: my_mser = lambda x,y: tf.reduce_mean(my_mse(x,y))
...:
...: y_true = tf.cast(np.random.randint(10, size=(6,1)), tf.float32)
...: y_pred = tf.cast(np.random.randint(10, size=(6,1)), tf.float32)
...:
...: i = tf.keras.layers.Input(shape=(1,))
...: o = tf.keras.layers.Dense(32)(i)
...: m = tf.keras.Model(inputs=i, outputs=o)
...:
...: m_pred = m(y_pred)
...:
...: for a,b in [(tf_mse, tf_mser), (my_mse, my_mser)]:
...: print(f'{a(y_true, y_pred).numpy()} -> {b(y_true, y_pred)}')
...:
...: for a,b in [(tf_mse, tf_mser), (my_mse, my_mser)]:
...: print(f'{a(y_true, m_pred).numpy()} -> {b(y_true, m_pred)}')
[ 9. 4. 4. 9. 16. 4.] -> 7.666666507720947
[ 9. 4. 4. 9. 16. 4.] -> 7.666666507720947
[20.47 30.15 19.62 9. 12.8 19.62] -> 18.608726501464844
[20.47 30.15 19.62 9. 12.8 19.62] -> 18.608726501464844
edit2:
ok, I get it. I get it. If I change the small_ds above to do this, everything works fine:
def small_ds():
in_t = tf.cast(np.random.randint(5, size=(24, 2)), tf.float32)
out_t = tf.expand_dims(tf.reduce_sum(in_t, axis=-1), -1)
return in_t, out_t
I think the secret to figuring this out was still in the first answer, though it wasn't clear to me exactly... The first mean in the real MSE work on channel -1, so if your shape is (24,), it's always going to reduce it -- regardless of the reduction setting.
By adding the expand_dims, my poor man's recreation of mse works just like the native one.
uj5u.com熱心網友回復：
根據docs，MeanSquaredError損失函式具有引數reduction，losses_utils.ReductionV2.AUTO默認設定為。現在這意味著：
減少選項將由使用背景關系確定。對于幾乎所有情況，這默認為 SUM_OVER_BATCH_SIZE。
所以我認為這取決于您使用的縮減方法和批量大小。嘗試small_ds()像這樣改變你的方法：
def small_ds():
in_t = tf.cast(np.random.randint(5, size=(1, 2)), tf.float32)
out_t = tf.reduce_sum(in_t, axis=-1)
return in_t, out_t
您會注意到批量大小 1 的結果是相同的：
Input shape: (1, 2)
step=0 theirs= 30.9056 mine= 30.9056
step=5 theirs= 21.4109 mine= 21.4109
step=10 theirs= 13.2141 mine= 13.2141
....
step=75 theirs= 0.0004 mine= 0.0004
step=80 theirs= 0.0055 mine= 0.0055
step=85 theirs= 0.0054 mine= 0.0054
step=90 theirs= 0.0015 mine= 0.0015
step=95 theirs= 0.0000 mine= 0.0000
示例reduction='none'：
y_true = tf.constant([[0., 2.], [0., 0.]])
y_pred = tf.constant([[3., 1.], [2., 5.]])
tf_mse = tf.keras.losses.MeanSquaredError(reduction='none')
print(tf_mse(y_true, y_pred).numpy())
my_mse = tf.reduce_mean(tf.square(y_true-y_pred))
print(my_mse)
'''
[ 5. 14.5]
tf.Tensor(9.75, shape=(), dtype=float32)
'''
并與tf.reduce_mean：
y_true = tf.constant([[0., 2.], [0., 0.]])
y_pred = tf.constant([[3., 1.], [2., 5.]])
tf_mse = tf.keras.losses.MeanSquaredError(reduction='none')
print(tf.reduce_mean(tf_mse(y_true, y_pred).numpy()))
my_mse = tf.reduce_mean(tf.square(y_true-y_pred))
print(my_mse)
'''
tf.Tensor(9.75, shape=(), dtype=float32)
tf.Tensor(9.75, shape=(), dtype=float32)
'''
并與reduction='sum'：
y_true = tf.constant([[0., 2.], [0., 0.]])
y_pred = tf.constant([[3., 1.], [2., 5.]])
tf_mse = tf.keras.losses.MeanSquaredError(reduction='sum')
print(tf_mse(y_true, y_pred).numpy())
my_mse = tf.reduce_mean(tf.square(y_true-y_pred))
print(my_mse)
'''
19.5
tf.Tensor(9.75, shape=(), dtype=float32)
'''
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/341014.html
標籤：tensorflow