我注意到 KL-Divergence 項 KL(Q(x)||P(x)) 在使用時的計算方式不同
mean(Q(x)*(log Q(x) - log P(x)))
對比
torch.distributions.kl_divergence(Q, P)
在哪里
Q = torch.distributions.Normal(some mean, some sigma)
P = torch.distributions.Normal(0, 1)
當我繪制 KL 散度損失時，我得到這兩個相似但不同的圖： 這里
誰能指出造成這種差異的原因是什么？
完整代碼如下：
import numpy as np
import torch
import torch.distributions as dist
import matplotlib.pyplot as plt
def kl_1(log_qx, log_px):
"""
inputs: [B, z_dim] torch
"""
return (log_qx.exp() * (log_qx-log_px)).mean()
# ground-truth (target) P(x)
P = dist.Normal(0, 1)
mus = np.arange(-5, 5, 0.1)
sigma = 1
N = 100
kls = {"1": [], "2": []}
for mu in mus:
# prediction (current) Q(x)
Q = dist.Normal(mu, sigma)
# sample from Q
qx = Q.sample((N,))
# log prob
log_qx = Q.log_prob(qx)
log_px = P.log_prob(qx)
# kl 1
kl1 = kl_1(log_qx, log_px)
kls['1'].append(kl1.numpy())
# kl 2
kl2 = dist.kl_divergence(Q, P)
kls['2'].append(kl2.numpy())
plt.figure()
plt.scatter(mus, kls['1'], label="Q*(logQ-logP)")
plt.scatter(mus, kls['2'], label="kl_divergence")
plt.xlabel("mean of Q(x)")
plt.ylabel("computed KL Divergence")
plt.legend()
plt.show()
uj5u.com熱心網友回復：
如果您從 上的積分計算期望值，則您的樣本按概率密度加權dx。如果您使用來自給定分布的樣本，那么您可以直接將期望值近似為平均值，這對應于 上的積分d p(x)。
import numpy as np
import torch
import torch.distributions as dist
import matplotlib.pyplot as plt
def kl_1(log_qx, log_px):
"""
inputs: [B, z_dim] torch
"""
return (log_qx-log_px).mean()
# ground-truth (target) P(x)
P = dist.Normal(0, 1)
mus = np.arange(-5, 5, 0.1)
sigma = 1
N = 100
kls = {"1": [], "2": []}
for mu in mus:
# prediction (current) Q(x)
Q = dist.Normal(mu, sigma)
# sample from Q
qx = Q.sample((N,))
# log prob
log_qx = Q.log_prob(qx)
log_px = P.log_prob(qx)
# kl 1
kl1 = kl_1(log_qx, log_px)
kls['1'].append(kl1.numpy())
# kl 2
kl2 = dist.kl_divergence(Q, P)
kls['2'].append(kl2.numpy())
plt.figure()
plt.scatter(mus, kls['1'], label="Q*(logQ-logP)")
plt.scatter(mus, kls['2'], label="kl_divergence")
plt.xlabel("mean of Q(x)")
plt.ylabel("computed KL Divergence")
plt.legend()
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/479918.html