我有一個玩具模型：
import torch
import torch.nn as nn
import torch.optim as optim
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.fc1 = nn.Linear(1, 2)
self.fc2 = nn.Linear(2, 3)
self.fc3 = nn.Linear(3, 1)
def forward(self, x):
x = self.fc1(x)
x = torch.relu(x)
x = torch.relu(self.fc2(x))
x = self.fc3(x)
return x
net = Model()
opt = optim.Adam(net.parameters())
訓練回圈是
features = torch.rand((3,1))
for i in range(10):
opt.zero_grad()
out = net(features)
loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
loss.backward()
opt.step()
如何檢查是否有任何漸變是 nan？也就是說，如果只有 1 個漸變是 nan print something/break
偽代碼：
for i in range(10):
opt.zero_grad()
out = net(features)
loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
loss.backward()
if_gradients_nan:
print("NAN")
opt.step()
uj5u.com熱心網友回復：
您可以檢查如下。這種方法只檢查關于模型引數的梯度。它不看中間梯度，實際上，那些中間梯度在沒有引數loss.backward()的情況下被呼叫后就不存在了。出于演示目的，我將 first的retain_graph=True輸出乘以，使得一些梯度變為.torch.relu(x)float("inf")nan
...
def forward(self, x):
x = self.fc1(x)
x = torch.relu(x) * float("inf")
x = torch.relu(self.fc2(x))
x = self.fc3(x)
return x
...
loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
loss.backward()
for name, param in net.named_parameters():
print(name, torch.isnan(param.grad))
opt.step()
這列印
fc1.weight tensor([[False],
[False]])
fc1.bias tensor([False, False])
fc2.weight tensor([[True, True],
[True, True],
[True, True]])
fc2.bias tensor([True, True, True])
fc3.weight tensor([[True, True, True]])
fc3.bias tensor([True])
fc1.weight tensor([[False],
[False]])
fc1.bias tensor([False, False])
fc2.weight tensor([[True, True],
[True, True],
[True, True]])
...
要檢查是否有任何漸變nan，您可以使用
for name, param in net.named_parameters():
if torch.isnan(param.grad).any():
print("nan gradient found")
raise SystemExit
轉載請註明出處，本文鏈接：https://www.uj5u.com/net/491852.html
上一篇：按特定班級數劃分訓練測驗