高斯混合模型GMM（Gaussian Mixture Model）
- 1. 模型介紹
- 2. 極大似然估計MLE（Maximum Likelihood Estimate）
- 3. EM求解（Expectation-Maximization Algorithm）
- (1) EM演算法（期望最大演算法）公式與收斂性
- (2) E-step
- (3) M-step
- 總結
1. 模型介紹
高斯——高斯分布
從概率密度估計的角度來看 ，從幾何角度來看，加權（
α
\alpha
α）平均→多個高斯分布疊加而成
p
(
x
)
=
∑
k
=
1
k
α
k
N
(
μ
k
,
Σ
k
)
,
∑
k
=
1
k
α
k
=
1
(1)
p(x)=\sum_{k=1}^{k} \alpha_{k} N\left(\mu_{k}, \Sigma_{k}\right), \sum_{k=1}^{k} \alpha_{k}=1\tag{1}
p(x)=k=1∑k?αk?N(μk?,Σk?),k=1∑k?αk?=1(1)
∑
α
k
=
1
\sum \alpha_{k}=1
∑αk?=1
上式中
α
k
\alpha_k
αk?為權重
從混合（生成）模型的角度來看（生成模型）
x：observed variable
z：latent variable
z表示對應的樣本x是屬于哪一個高斯分布，這是一個離散的隨機變數
生成程序，概率圖（有向圖）如下：（觀測圖用陰影表示）
聯合概率密度可轉化為乘法的形式：
p
(
x
)
=
∑
z
p
(
x
,
z
)
=
∑
k
=
1
k
p
(
x
,
z
=
c
k
)
=
∑
k
=
1
k
p
(
z
=
c
k
)
?
p
(
x
∣
z
=
c
k
)
=
∑
k
=
1
K
p
k
?
N
(
x
∣
μ
k
Σ
k
)
(2)
\begin{aligned} p(x) &=\sum_{z} p(x, z) \\ &=\sum_{k=1}^{k} p\left(x, z=c_{k}\right) \\ &=\sum_{k=1}^{k} p\left(z=c_{k}\right) \cdot p\left(x \mid z=c_{k}\right) \\ &=\sum_{k=1}^{K} p_{k} \cdot N\left(x \mid \mu_{k} \Sigma_{k}\right) \end{aligned}\tag{2}
p(x)?=z∑?p(x,z)=k=1∑k?p(x,z=ck?)=k=1∑k?p(z=ck?)?p(x∣z=ck?)=k=1∑K?pk??N(x∣μk?Σk?)?(2)
上式中
p
k
p_k
pk?為概率值，可見與式(1)相同
2. 極大似然估計MLE（Maximum Likelihood Estimate）
X：observed data （
X
=
(
x
1
,
x
2
,
.
.
.
,
x
N
)
X=(x_1,x_2,...,x_N)
X=(x1?,x2?,...,xN?)）
(X,Z)：complete data
θ
\theta
θ：parameter （
θ
\theta
θ={
p
1
,
p
2
,
.
.
.
,
p
k
,
μ
1
,
μ
2
,
.
.
.
,
μ
k
,
Σ
1
,
Σ
2
,
.
.
.
,
Σ
k
p_1,p_2,...,p_k,\mu_1,\mu_2,...,\mu_k,\Sigma_1,\Sigma_2,...,\Sigma_k
p1?,p2?,...,pk?,μ1?,μ2?,...,μk?,Σ1?,Σ2?,...,Σk?}）
θ
^
M
L
E
=
arg
?
max
?
θ
log
?
P
(
x
)
\hat{\theta}_{MLE}=\arg \max _{\theta} \log P(x)
θ^MLE?=argθmax?logP(x)
樣本之間相互獨立，上式可寫為相乘的形式
θ
^
M
L
E
=
arg
?
max
?
θ
log
?
∏
i
=
1
N
P
(
x
i
)
=
arg
?
max
?
∑
i
=
1
N
log
?
P
(
x
i
)
\hat{\theta}_{MLE}=\arg \max _{\theta} \log \prod_{i=1}^{N} P\left(x_{i}\right)=\arg \max \sum_{i=1}^{N} \log P\left(x_{i}\right)
θ^MLE?=argθmax?logi=1∏N?P(xi?)=argmaxi=1∑N?logP(xi?)
將式(2)代入，得
θ
^
M
L
E
=
arg
?
max
?
θ
∑
i
=
1
N
log
?
∑
k
=
1
k
p
k
?
N
(
x
i
∣
μ
k
,
Σ
k
)
\hat{\theta}_{MLE}=\arg \max _{\theta} \sum_{i=1}^{N} \log \sum_{k=1}^{k} p_{k} \cdot N\left(x_{i} \mid \mu_{k}, \Sigma_{k}\right)
θ^MLE?=argθmax?i=1∑N?logk=1∑k?pk??N(xi?∣μk?,Σk?)
log里面是一個連加的形式，直接用MLE求解GMM，得不到決議解，一般使用EM求出 θ \theta θ
3. EM求解（Expectation-Maximization Algorithm）
(1) EM演算法（期望最大演算法）公式與收斂性
主要用以估計還有隱變數的引數估計
MLE：
P
(
x
∣
θ
)
P(x|\theta)
P(x∣θ)
θ
M
L
E
=
argmax
?
log
?
P
(
x
∣
θ
)
\theta_{MLE}=\operatorname{argmax} \log P(x \mid \theta)
θMLE?=argmaxlogP(x∣θ)
log-likelihood
EM公式如下：
θ
(
t
+
1
)
=
arg
?
max
?
θ
∫
z
log
?
p
(
x
,
z
∣
θ
)
?
p
(
z
∣
x
,
θ
(
t
)
)
d
z
(3)
\theta^{(t+1)}=\arg \max _{\theta} \int_{z} \log p(x, z \mid \theta) \cdot p\left(z \mid x, \theta^{(t)}\right) d z\tag{3}
θ(t+1)=argθmax?∫z?logp(x,z∣θ)?p(z∣x,θ(t))dz(3)
第一項為對數聯合概率，第二項為后驗概率
E
z
∣
x
θ
(
t
)
[
log
?
P
(
x
,
z
∣
θ
)
]
E_{z \mid x \theta^{(t)}}[\log P(x, z \mid \theta)]
Ez∣xθ(t)?[logP(x,z∣θ)]
對于收斂性，需要證明：
log
?
P
(
x
∣
θ
(
t
)
)
?
log
?
P
(
x
∣
θ
(
t
+
1
)
)
\log P\left( x| \theta^{(t)}\right) \leqslant \log P\left(x \mid \theta^{(t+1)}\right)
logP(x∣θ(t))?logP(x∣θ(t+1))
對于
log
?
P
(
x
∣
θ
)
=
log
?
P
(
x
,
z
∣
θ
)
?
log
?
P
(
z
∣
x
,
θ
)
\log P(x \mid \theta)=\log P(x, z \mid \theta)-\log P(z \mid x, \theta)
logP(x∣θ)=logP(x,z∣θ)?logP(z∣x,θ)
對Q進行積分：
左邊
=
∫
z
p
(
z
∣
x
,
θ
(
t
)
)
?
log
?
P
(
x
∣
θ
)
d
z
=
log
?
P
(
x
∣
θ
)
∫
z
P
(
z
∣
x
,
θ
(
t
)
)
d
z
?
1
=
log
?
P
(
x
∣
θ
)
\begin{aligned} \text { 左邊 }&=\int_{z} p\left(z \mid x, \theta^{(t)}\right) \cdot \log P(x \mid \theta) d z\\ &=\log P(x \mid \theta) \underbrace{\int_{z} P\left(z \mid x, \theta^{(t)}\right) d z}_{1}\\ &=\log P(x \mid \theta) \end{aligned}
左邊 ?=∫z?p(z∣x,θ(t))?logP(x∣θ)dz=logP(x∣θ)1
∫z?P(z∣x,θ(t))dz??=logP(x∣θ)?
右邊
=
∫
z
P
(
Z
∣
x
,
θ
(
t
)
)
?
log
?
P
(
x
,
z
∣
θ
)
?
Q
(
θ
,
θ
(
t
)
)
d
z
?
∫
z
P
(
z
∣
x
,
θ
(
t
)
)
?
log
?
P
(
Z
∣
x
,
θ
)
d
z
?
H
(
θ
,
θ
(
t
)
)
\begin{aligned} \text { 右邊 }&= \underbrace{\int_{z} P\left(Z \mid x, \theta^{(t)}\right) \cdot \log P(x, z \mid \theta)}_{Q\left(\theta, \theta^{(t)}\right)} d z- \underbrace{\int_{z} P\left(z \mid x, \theta^{(t)}\right) \cdot \log P(Z \mid x, \theta) d z}_{H\left(\theta, \theta^{(t)}\right)} \end{aligned}
右邊 ?=Q(θ,θ(t))
∫z?P(Z∣x,θ(t))?logP(x,z∣θ)??dz?H(θ,θ(t))
∫z?P(z∣x,θ(t))?logP(Z∣x,θ)dz???
對于第一項已經得證：
Q
(
θ
(
t
+
1
)
,
θ
(
t
)
)
?
Q
(
θ
(
t
)
,
θ
(
t
)
)
\begin{aligned} Q\left(\theta^{(t+1)}, \theta^{(t)}\right) & \geqslant Q\left(\theta^{(t)}, \theta^{(t)}\right) \\ \end{aligned}
Q(θ(t+1),θ(t))??Q(θ(t),θ(t))?
下面僅需證明：
H
(
θ
(
t
+
1
)
,
θ
(
t
)
)
?
H
(
θ
(
t
)
,
θ
(
t
)
)
H\left(\theta^{(t+1)}, \theta^{(t)}\right) \leqslant H\left(\theta^{(t)}, \theta^{(t)}\right)
H(θ(t+1),θ(t))?H(θ(t),θ(t))
直接相減得：
H
(
θ
(
t
+
1
)
,
θ
(
t
)
)
?
H
(
θ
(
t
)
?
θ
(
t
)
)
=
∫
z
P
(
z
∣
x
,
θ
(
t
)
)
?
log
?
P
(
z
∣
x
θ
(
t
+
1
)
)
d
z
?
∫
z
P
(
z
∣
x
,
θ
(
t
)
)
?
log
?
P
(
z
∣
x
,
θ
(
t
)
)
d
z
=
∫
z
P
(
z
∣
x
,
θ
(
t
)
)
?
log
?
P
(
z
∣
x
,
θ
(
+
+
1
)
)
p
(
z
∣
x
,
θ
(
1
)
)
d
z
E
[
log
?
x
]
?
log
?
E
[
x
]
?
log
?
∫
z
p
(
z
∣
x
,
θ
(
t
+
1
)
)
d
z
?
1
=
log
?
1
=
0
\begin{aligned} & H\left(\theta^{(t+1)}, \theta^{(t)}\right)-H\left(\theta^{(t)} \cdot \theta^{(t)}\right) \\ =& \int_{z} P\left(z \mid x, \theta^{(t)}\right) \cdot \log P\left(z \mid x \theta^{(t+1)}\right) d z \\ -& \int_{z} P\left(z \mid x, \theta^{(t)}\right) \cdot \log P\left(z \mid x, \theta^{(t)}\right) d z \\ =& \int_{z} P\left(z \mid x, \theta^{(t)}\right) \cdot \log \frac{P\left(z \mid x, \theta^{(++1)}\right)}{p\left(z \mid x, \theta^{(1)}\right)} d z \\ & E[\log x] \leqslant \log E[x] \\ \leqslant & \log \underbrace{\int_{z} p\left(z \mid x, \theta^{(t+1)}\right) d z}_{1}=\log 1=0 \end{aligned}
=?=??H(θ(t+1),θ(t))?H(θ(t)?θ(t))∫z?P(z∣x,θ(t))?logP(z∣xθ(t+1))dz∫z?P(z∣x,θ(t))?logP(z∣x,θ(t))dz∫z?P(z∣x,θ(t))?logp(z∣x,θ(1))P(z∣x,θ(++1))?dzE[logx]?logE[x]log1
∫z?p(z∣x,θ(t+1))dz??=log1=0?
最后一步不是很懂
(2) E-step
E
M
:
θ
(
t
+
1
)
=
arg
?
max
?
E
z
∣
x
θ
(
i
)
[
log
?
P
(
x
,
z
∣
θ
)
]
=
arg
?
max
?
(
Q
(
θ
,
θ
t
)
)
\begin{aligned} E M: \theta^{(t+1)}&=\arg \max E_{{z|x } \theta^{(i)}}[\log P(x, z \mid \theta)] \\ &=\arg \max(Q(\theta,\theta^{t})) \end{aligned}
EM:θ(t+1)?=argmaxEz∣xθ(i)?[logP(x,z∣θ)]=argmax(Q(θ,θt))?
這是一個迭代的方式，逐步去逼近和的最大值，由公式(3)
Q
(
θ
,
θ
(
t
)
)
=
∫
z
log
?
P
(
X
,
z
∣
θ
)
?
P
(
z
∣
X
,
θ
(
t
)
)
d
z
=
∑
Z
log
?
∏
i
=
1
N
P
(
x
i
,
z
i
∣
θ
)
?
?
∏
i
=
1
N
P
(
z
i
∣
x
i
,
θ
(
t
)
)
=
∑
z
1
,
z
2
,
z
N
∑
i
=
1
N
log
?
P
(
x
i
,
z
i
∣
θ
)
?
∏
i
=
1
N
P
(
z
i
∣
x
i
,
θ
(
t
)
)
=
∑
z
1
,
z
2
,
z
N
[
log
?
P
(
x
1
,
z
1
∣
θ
)
+
log
?
P
(
x
2
,
z
2
∣
θ
)
+
?
+
log
?
P
(
x
N
,
z
N
∣
θ
)
]
∏
i
=
1
N
P
(
z
i
∣
x
i
,
θ
(
t
)
)
\begin{aligned} &Q\left(\theta, \theta^{(t)}\right)=\int_{z} \log P(X, z \mid \theta) \cdot P\left(z \mid X, \theta^{(t)}\right) d z\\ &=\sum_{Z} \underbrace{\log \prod_{i=1}^{N} P\left(x_{i}, z_{i} \mid \theta\right)} \cdot \prod_{i=1}^{N} P\left(z_{i} \mid x_{i}, \theta^{(t)}\right)\\ &=\sum_{z_1, z_{2}, z_{N}} \sum_{i=1}^{N} \log P\left(x_{i}, z_{i} \mid \theta\right) \cdot \prod_{i=1}^{N} P\left(z_{i} \mid x_{i}, \theta^{(t)}\right)\\ &=\sum_{z_1, z_{2},z_{N}}\left[\log P\left(x_{1}, z_{1} \mid \theta\right)+\log P\left(x_{2}, z_{2} \mid \theta\right)+\cdots+\log P\left(x_{N}, z_{N} \mid \theta\right)\right] \prod_{i=1}^{N} P\left(z_{i} \mid x_{i}, \theta^{(t)}\right) \end{aligned}
?Q(θ,θ(t))=∫z?logP(X,z∣θ)?P(z∣X,θ(t))dz=Z∑?
logi=1∏N?P(xi?,zi?∣θ)??i=1∏N?P(zi?∣xi?,θ(t))=z1?,z2?,zN?∑?i=1∑N?logP(xi?,zi?∣θ)?i=1∏N?P(zi?∣xi?,θ(t))=z1?,z2?,zN?∑?[logP(x1?,z1?∣θ)+logP(x2?,z2?∣θ)+?+logP(xN?,zN?∣θ)]i=1∏N?P(zi?∣xi?,θ(t))?
因為：
P
(
x
,
z
)
=
P
(
z
)
?
p
(
x
∣
z
)
=
p
z
?
N
(
x
∣
μ
z
,
Σ
z
)
\begin{aligned} P(x, z) &=P(z) \cdot p(x \mid z) \\ &=p_{z} \cdot N\left(x \mid \mu_{z}, \Sigma_{z}\right) \\ \end{aligned}
P(x,z)?=P(z)?p(x∣z)=pz??N(x∣μz?,Σz?)?
P
(
z
∣
x
)
=
p
(
x
,
z
)
p
(
x
)
=
p
z
?
N
(
x
∣
μ
z
Σ
z
)
∑
k
=
1
k
p
k
?
N
(
x
∣
μ
k
,
∑
k
)
P(z \mid x) =\frac{p(x, z)}{p(x)}=\frac{p_{z} \cdot N\left(x \mid \mu_{z} \Sigma_{z}\right)}{\sum_{k=1}^{k} p_{k} \cdot N\left(x \mid \mu_{k}, \sum_{k}\right)}
P(z∣x)=p(x)p(x,z)?=∑k=1k?pk??N(x∣μk?,∑k?)pz??N(x∣μz?Σz?)?
∑
z
1
,
z
2
,
z
N
log
?
P
(
x
1
,
z
1
∣
θ
)
?
∏
i
=
1
N
P
(
z
i
∣
x
i
,
θ
(
t
)
)
=
∑
z
1
log
?
P
(
x
1
,
z
1
∣
θ
)
?
p
(
z
1
∣
x
1
,
θ
(
t
)
)
\sum_{z_{1},z_{2},z_N} {\log P\left(x_{1}, z_{1} \mid \theta\right) \cdot \prod_{i=1}^{N} P\left(z_{i} \mid x_{i}, \theta^{(t)}\right)}=\sum_{z_{1}} \log P\left(x_{1}, z_{1} \mid \theta\right) \cdot p\left(z_{1} \mid x_{1}, \theta^{(t)}\right)
z1?,z2?,zN?∑?logP(x1?,z1?∣θ)?i=1∏N?P(zi?∣xi?,θ(t))=z1?∑?logP(x1?,z1?∣θ)?p(z1?∣x1?,θ(t))
上式：
=
∑
z
1
log
?
P
(
x
1
,
z
1
∣
θ
)
?
P
(
z
1
∣
x
1
,
θ
(
t
)
)
+
?
+
∑
Z
N
log
?
P
(
x
N
,
z
N
∣
θ
)
?
p
(
z
N
∣
x
N
,
θ
(
t
)
)
=
∑
i
=
1
N
∑
Z
i
log
?
P
(
x
i
,
z
i
∣
θ
)
?
P
(
z
i
∣
x
i
,
θ
(
t
)
)
=
∑
i
=
1
N
∑
z
i
log
?
[
p
z
i
?
N
(
x
i
∣
μ
z
i
,
Σ
z
i
)
]
?
p
z
i
?
N
(
x
i
∣
μ
z
i
,
(
t
)
Σ
z
i
(
t
)
)
∑
k
=
1
k
p
k
(
t
)
?
N
(
x
i
∣
μ
k
(
t
)
Σ
k
(
t
)
)
(4)
\begin{aligned} &=\sum_{z_{1}} \log P\left(x_{1}, z_1 \mid \theta\right) \cdot P\left(z_{1} \mid x_{1}, \theta^{(t)}\right)+\cdots+\sum_{Z_N} \log P\left(x_{N} ,z_{N} \mid \theta\right) \cdot p\left(z_{N} \mid x_{N}, \theta^{(t)}\right)\\ &=\sum_{i=1}^{N} \sum_{Z_i} \log P\left(x_{i}, z_{i} \mid \theta\right) \cdot P\left(z_{i} \mid x_{i}, \theta^{(t)}\right)\\\tag{4} &=\sum_{i=1}^{N} \sum_{z_{i}} \log [p_{z_{i}} \cdot N\left(x_{i} \mid \mu_{z_i}, \Sigma_{z_i}\right)] \cdot \frac{p_{z_{i}} \cdot N\left(x_{i} \mid \mu_{z_i,}^{(t)} \Sigma_{z_i}^{(t)}\right)}{\sum_{k=1}^{k} p_{k}^{(t)} \cdot N\left(x_{i} \mid \mu_{k}^{(t)} \Sigma_{k}^{(t)}\right)} \end{aligned}
?=z1?∑?logP(x1?,z1?∣θ)?P(z1?∣x1?,θ(t))+?+ZN?∑?logP(xN?,zN?∣θ)?p(zN?∣xN?,θ(t))=i=1∑N?Zi?∑?logP(xi?,zi?∣θ)?P(zi?∣xi?,θ(t))=i=1∑N?zi?∑?log[pzi???N(xi?∣μzi??,Σzi??)]?∑k=1k?pk(t)??N(xi?∣μk(t)?Σk(t)?)pzi???N(xi?∣μzi?,(t)?Σzi?(t)?)??(4)
注意第二項：
μ
z
i
(
t
)
,
Σ
Z
i
(
t
)
\mu_{z_i}^{(t)}, \Sigma_{Z_i}^{(t)}
μzi?(t)?,ΣZi?(t)?
式(4)可繼續寫：
=
∑
i
=
1
N
∑
z
i
log
?
[
p
z
i
?
N
(
x
i
∣
μ
ε
i
,
Σ
z
i
)
]
?
P
(
z
i
∣
x
i
,
θ
(
t
)
)
=
∑
z
i
∑
i
=
1
N
log
?
p
z
i
?
N
(
x
i
∣
μ
z
i
,
Σ
z
i
)
?
P
(
z
i
∣
x
i
,
θ
(
t
)
)
=
∑
k
=
1
K
∑
i
=
1
N
log
?
[
p
k
?
N
(
x
i
∣
μ
k
,
Σ
k
)
]
P
(
z
i
=
c
k
∣
x
i
,
θ
(
t
)
)
=
∑
k
=
1
k
∑
i
=
1
N
[
log
?
p
k
+
log
?
N
(
x
i
∣
μ
k
Σ
k
)
]
?
P
(
z
i
=
c
k
∣
x
i
,
θ
(
t
)
)
\begin{aligned} &=\sum_{i=1}^{N} \sum_{z_{i}} \log \left[p_{z_{i}} \cdot N\left(x_{i} \mid \mu_{\varepsilon_{i}}, \Sigma_{z_{i}}\right)\right] \cdot P\left(z_{i} \mid x_{i}, \theta^{(t)}\right)\\ &=\sum_{z_i} \sum_{i=1}^{N} \log p_{z_{i}} \cdot N\left(x_{i} \mid \mu_{z_{i}}, \Sigma_{z_{i}}\right) \cdot P\left(z_{i} \mid x_{i}, \theta^{(t)}\right)\\ &=\sum_{k=1}^{K} \sum_{i=1}^{N} \log \left[p_{k} \cdot N\left(x_{i} \mid \mu_{k}, \Sigma_{k}\right)\right] P\left(z_{i}=c_{k} \mid x_{i}, \theta^{(t)}\right)\\ &=\sum_{k=1}^{k} \sum_{i=1}^{N}\left[\log p_{k}+\log N\left(x_{i} \mid \mu_{k} \Sigma_{k}\right)\right] \cdot P\left(z_{i}=c_{k} \mid x_{i}, \theta^{\left(t\right)}\right) \end{aligned}
?=i=1∑N?zi?∑?log[pzi???N(xi?∣μεi??,Σzi??)]?P(zi?∣xi?,θ(t))=zi?∑?i=1∑N?logpzi???N(xi?∣μzi??,Σzi??)?P(zi?∣xi?,θ(t))=k=1∑K?i=1∑N?log[pk??N(xi?∣μk?,Σk?)]P(zi?=ck?∣xi?,θ(t))=k=1∑k?i=1∑N?[logpk?+logN(xi?∣μk?Σk?)]?P(zi?=ck?∣xi?,θ(t))?
(3) M-step
θ
(
t
+
1
)
=
argmax
?
θ
Q
(
θ
,
θ
(
t
)
)
\theta^{(t+1)}=\underset{\theta}{\operatorname{argmax}} Q\left(\theta, \theta^{(t)}\right)
θ(t+1)=θargmax?Q(θ,θ(t))
下面以求
p
k
(
t
+
1
)
p_k^{(t+1)}
pk(t+1)?為例：
p
k
(
t
+
1
)
=
arg?max
?
p
k
∑
k
=
1
k
∑
i
=
1
N
log
?
p
k
?
P
(
Z
i
=
C
k
∣
x
i
,
θ
(
t
)
)
,
s.t
∑
k
=
1
K
p
k
=
1
p_{k}^{(t+1)}=\argmax_{p_{k}} \sum_{k=1}^{k} \sum_{i=1}^{N} \log p_{k} \cdot P\left(Z_{i}=C_{k} \mid x_{i}, \theta^{(t)}\right), \text { s.t } \sum_{k=1}^{K} p_{k}=1
pk(t+1)?=pk?argmax?k=1∑k?i=1∑N?logpk??P(Zi?=Ck?∣xi?,θ(t)), s.t k=1∑K?pk?=1
約束優化問題，使用拉格朗日乘子法，首先定義拉格朗日乘子
L
(
p
,
λ
)
=
∑
k
=
1
k
∑
i
=
1
N
log
?
p
k
?
P
(
Z
i
=
C
k
∣
x
i
,
θ
(
t
)
)
+
λ
(
∑
k
=
1
k
p
k
?
1
)
\mathcal{L}(p, \lambda)=\sum_{k=1}^{k} \sum_{i=1}^{N} \log p_{k} \cdot P\left(Z_{i}=C_{k} \mid x_{i}, \theta^{(t)}\right)+\lambda\left(\sum_{k=1}^{k} p_{k}-1\right)
L(p,λ)=k=1∑k?i=1∑N?logpk??P(Zi?=Ck?∣xi?,θ(t))+λ(k=1∑k?pk??1)
求偏導數令其為0：
?
f
?
p
k
=
∑
i
=
1
N
1
p
k
?
P
(
Z
i
=
C
k
∣
x
i
,
θ
(
t
)
)
+
λ
?
0
\frac{\partial \mathcal{f}}{\partial p_{k}}=\sum_{i=1}^{N} \frac{1}{p_{k}} \cdot P\left(Z_{i}=C_{k} \mid x_{i}, \theta^{(t)}\right)+\lambda \triangleq 0
?pk??f?=i=1∑N?pk?1??P(Zi?=Ck?∣xi?,θ(t))+λ?0
?
∑
i
=
1
N
P
(
z
i
=
C
R
∣
x
i
,
θ
(
t
)
)
+
p
k
i
λ
i
=
0
?
k
=
1
,
?
?
,
K
?
∑
i
=
1
N
∑
k
=
1
K
p
(
Z
i
=
C
k
∣
x
i
,
θ
(
t
)
?
1
)
+
∑
k
=
1
K
p
k
λ
?
1
=
0
?
N
+
λ
=
0
\begin{aligned} &\Rightarrow \sum_{i=1}^{N} P\left(z_{i}=C_{R} \mid x_{i}, \theta^{(t)}\right)+p_{k} i \lambda_{i}=0\\ &\Rightarrow{k=1, \cdots,K}{\Rightarrow} \underbrace{\sum_{i=1}^{N} \sum_{k=1}^{K} p\left(Z_{i}=C_{k} \mid x_{i}, \theta^{(t)}\right.}_{1})+\underbrace{\sum_{k=1}^{K} p_{k} \lambda}_{1}=0\\ &\Rightarrow N+\lambda=0 \end{aligned}
??i=1∑N?P(zi?=CR?∣xi?,θ(t))+pk?iλi?=0?k=1,?,K?1
i=1∑N?k=1∑K?p(Zi?=Ck?∣xi?,θ(t)??)+1
k=1∑K?pk?λ??=0?N+λ=0?
p
k
(
t
+
1
)
=
1
N
∑
i
=
1
N
P
(
Z
i
=
C
k
∣
x
i
,
θ
(
t
)
)
p
(
t
+
1
)
=
(
p
1
(
t
+
1
)
,
?
?
,
p
k
(
t
+
1
)
)
\begin{aligned} &p_{k}^{(t+1)}=\frac{1}{N} \sum_{i=1}^{N} P\left(Z_{i}=C_{k} \mid x_{i}, \theta^{(t)}\right)\\ &p^{(t+1)}=\left(p_{1}^{(t+1)}, \cdots, p_{k}^{(t+1)}\right) \end{aligned}
?pk(t+1)?=N1?i=1∑N?P(Zi?=Ck?∣xi?,θ(t))p(t+1)=(p1(t+1)?,?,pk(t+1)?)?
總結
基本了解了GMM模型是什么，以及如何通過EM演算法去求解，手推公式感覺很不友好，開始入門HMM
轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/301671.html
標籤：AI
上一篇：OpenCV-美食—鮮美濾鏡