我在復制這個Pytorch 教程時遇到了麻煩。
他們介紹的模型是：
class RNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(RNN, self).__init__()
self.hidden_size = hidden_size
self.i2h = nn.Linear(input_size hidden_size, hidden_size)
self.i2o = nn.Linear(input_size hidden_size, output_size)
self.softmax = nn.LogSoftmax(dim=1)
def forward(self, input, hidden):
combined = torch.cat((input, hidden), 1)
hidden = self.i2h(combined)
output = self.i2o(combined)
output = self.softmax(output)
return output, hidden
def initHidden(self, batch_size):
return torch.zeros(batch_size, self.hidden_size, dtype=torch.float32, requires_grad=True)
該模型再現了 RNN 單元內發生的情況。
在編碼時，我遇到了模型內部的漸變問題。
重現該問題的代碼如下：
import torch
import torch.nn as nn
# Toy data to reproduce the issue
toy_data_batch = torch.tensor([[0, 1], [1, 0], [1, 0]], dtype=torch.float32)
toy_label_batch = torch.tensor([2, 0, 3], dtype=torch.long)
class RNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(RNN, self).__init__()
self.hidden_size = hidden_size
self.i2h = nn.Linear(input_size hidden_size, hidden_size)
self.i2o = nn.Linear(input_size hidden_size, output_size)
self.softmax = nn.LogSoftmax(dim=1)
def forward(self, input, hidden):
combined = torch.cat((input, hidden), 1)
hidden = self.i2h(combined)
output = self.i2o(combined)
output = self.softmax(output)
return output, hidden
def initHidden(self, batch_size):
return torch.zeros(batch_size, self.hidden_size, dtype=torch.float32, requires_grad=True)
# Model initialization
input_size = 2
hidden_size = 2
output_size = 4 # Targets in {0, 1, 2, 3}
batch_size = 3 # 3 data points in the batch
learning_rate = 5e-3
rnn = RNN(input_size, hidden_size, output_size)
hidden = rnn.initHidden(batch_size) # init hidden layer with zeros
# Negative log likelihood as it is classification
criterion = nn.NLLLoss()
# Forward pass
output, hidden = rnn(toy_data_batch, hidden)
#output, hidden = rnn(toy_data_batch, hidden) ### BUG: if I remove the comment here, It works
# Loss computation
loss = criterion(output, toy_label_batch)
# Backward pass
loss.backward()
print(rnn.i2o.weight.grad) # This one is fine
print(rnn.i2h.weight.grad) # This one isn't (has type None)
# This will fail, because of the None gradient
for weight in rnn.parameters():
weight.data.add_(weight.grad.data, alpha=-learning_rate)
輸出是：
tensor([[-0.1892, 0.0462, 0.0000, 0.0000],
[ 0.1274, 0.1133, 0.0000, 0.0000],
[ 0.1455, -0.2525, 0.0000, 0.0000],
[-0.0837, 0.0930, 0.0000, 0.0000]])
None
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-3-5f02113fddf6> in <module>
54 # This will fail, because of the None gradient
55 for weight in rnn.parameters():
---> 56 weight.data.add_(weight.grad.data, alpha=-learning_rate)
AttributeError: 'NoneType' object has no attribute 'data'
我注意到如果我取消注釋該行
output, hidden = rnn(toy_data_batch, hidden)
#output, hidden = rnn(toy_data_batch, hidden) ### BUG: if I remove the comment here, It works
一切正常，沒有問題。在我看來，隱藏變數的初始化存在問題。當我打開引數“requires_grad”時，我不知道該怎么做。
提前感謝您，任何幫助將不勝感激
uj5u.com熱心網友回復：
self.i2h沒有梯度，因為它沒有在模型的第一步中使用。當您反向傳播時，您的模型僅self.i2o在第一階段使用，因此self.i2h對輸出沒有影響。但是，當您進入第二步時，它利用了hidden已使用 計算的 a self.i2h，因此通過該層存在可追蹤的梯度。
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/416632.html
標籤：