學習Coursera上Mathematics for Machine Learning Specialization后所做的筆記與整理

第一部分 Linear Algebra 線性代數

1. Vector operations 矢量運算

• commutative 交換律: r + s = s + r \text{commutative 交換律:} \quad r + s = s + r commutative 交換律:r+s=s+r
• 2 r = r + r 2r = r + r 2r=r+r
• ∥ r ∥ 2 = ∑ i r i 2 \|r\|^2 = \sum_{i} r_i^2 ∥r∥2=∑i?ri2?

1.1 dot or inner product 點積/數量積/內積

點積是一種特殊的內積
r ? s = ∑ i r i s i r \cdot s = \sum_{i} r_i s_i r?s=i∑?ri?si?

• commutative 交換律: r ? s = s ? r \text{commutative 交換律:} \quad r \cdot s = s \cdot r commutative 交換律:r?s=s?r
• distributive 分配律: r ? ( s + t ) = r ? s + r ? t \text{distributive 分配律:} \quad r \cdot (s + t) = r \cdot s + r \cdot t distributive 分配律:r?(s+t)=r?s+r?t
• associative 結合律 r ? ( a s ) = a ( r ? s ) \text{associative 結合律} \quad r \cdot (a s) = a(r \cdot s) associative 結合律r?(as)=a(r?s)
• r ? r = ∥ r ∥ 2 r \cdot r = \|r\|^2 r?r=∥r∥2
• r ? s = ∥ r ∥ ∥ s ∥ cos ? θ r \cdot s = \|r\| \|s\| \cos \theta r?s=∥r∥∥s∥cosθ

1.2 scalar and vector projection 投影

• scalar projection 投影/標量投影
  例：向量s在向量r上的投影 r ? s ∥ r ∥ \frac{r \cdot s}{\|r\|} ∥r∥r?s?
• vector projection 矢量投影
  例：向量s在向量r上的投影 r ? s r ? r r \frac{r \cdot s} {r \cdot r} r r?rr?s?r

2. basis 基

A basis is a set of n n n vectors that:

• are not linear combinations of each other
• span the space
  The Space is then n-dimensional.

在線性空間 V V V中，如果存在 n n n個元素 a 1 , a 2 , … , a n a_1,a_2,\dots,a_n a1?,a2?,…,an?，滿足：

• a 1 , a 2 , … , a n a_1,a_2,\dots,a_n a1?,a2?,…,an?線性無關；
• V V V中任一元素 a a a總可由 a 1 , a 2 , … , a n a_1,a_2,\dots,a_n a1?,a2?,…,an?線性表示，

那么， a 1 , a 2 , … , a n a_1,a_2,\dots,a_n a1?,a2?,…,an?就稱為線性空間 V V V的一個基， n n n稱為線性空間 V V V的維數，只含有一個零元素的線性空間沒有基，規定它的維數為0.
維數為 n n n的線性空間稱為 n n n維線性空間，記作 V n V_n Vn?，
（同濟大學線性代數第五版第六章第二節）

3. Matrices 矩陣

由 m × n m \times n m×n個數 a i j ( i = 1 , 2 , ? ? , m ; j = 1 , 2 , … , n ) a_{ij}(i=1,2,\cdots,m;j=1,2,\dots,n) aij?(i=1,2,?,m;j=1,2,…,n)排成的 m m m行 n n n列的數表稱為 m m m行 n n n列矩陣，簡稱 m × n m \times n m×n矩陣，記作
A = ( a 11 a 12 ? a 1 n a 21 a 22 ? a 2 n ? ? ? ? a m 1 a m 2 ? a m n ) A = \begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix} A=??????a11?a21??am1??a12?a22??am2???????a1n?a2n??amn????????
（同濟大學線性代數第五版第二章第一節）

矩陣與向量相乘

[ a b c d ] [ e f ] = [ a e + b f c e + d f ] \begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix} e \\ f \end{bmatrix} = \begin{bmatrix} ae + bf \\ ce + df \end{bmatrix} [ac?bd?][ef?]=[ae+bfce+df?]

• 向量與矩陣相乘可以理解為: 向量 r r r經過矩陣A變換為 r ′ A r = r ′ r' \quad Ar=r' r′Ar=r′
• A ( n r ) = n ( A r ) = n r ′ A(nr) = n(Ar) = nr' A(nr)=n(Ar)=nr′
• 分配律 A ( r + s ) = A r + A s \quad A(r + s) = Ar + As A(r+s)=Ar+As
• Identity 單位矩陣
  I = [ 1 0 0 1 ] I = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} I=[10?01?]
• clockwise rotation by θ \theta θ 順時針旋轉 θ \theta θ角度
  [ cos ? θ sin ? θ ? sin ? θ cos ? θ ] \begin{bmatrix} \cos \theta & \sin \theta \\ - \sin \theta & \cos \theta \end{bmatrix} [cosθ?sinθ?sinθcosθ?]
• determinant of 2×2 matrix 行列式
  ∣ A ∣ = d e t A = d e t [ a b c d ] = a d ? b d |A| = det A= det \begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bd ∣A∣=detA=det[ac?bd?]=ad?bd
• inverse of 2×2 matrix 逆矩陣
  [ a b c d ] ? 1 = 1 a d ? b c [ d ? b ? c a ] \begin{bmatrix} a & b \\ c & d \end{bmatrix}^{-1} = \frac {1} {ad -bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix} [ac?bd?]?1=ad?bc1?[d?c??ba?]
• summation convention for multiplying matrices A A A and B B B
  A B = C AB=C AB=C
  c i k = a b i k = ∑ j a i j b j k c_{ik} = ab_{ik} = \sum_j a_{ij}b_{jk} cik?=abik?=j∑?aij?bjk?

4. change of basis 基變換/坐標變換

Change from an original basis to a new, pried basis. The columns of the transformation matrix B B B are the new basis vectors in the original coordinate system. So
B r ′ = r Br' = r Br′=r
where r ′ r' r′ is the vector in the B B B-basis, and r r r is the vector in the original basis. Or;
r ′ = B ? 1 r r' = B^{-1}r r′=B?1r
If a matrix A A A is orthonormal (all the columns are of unit size and orthogonal to each other) then
矩陣 A A A是正交矩陣（正交陣）的充分必要條件是 A A A等列向量都是單位向量，且兩兩正交，
A T = A ? 1 A^T = A^{-1} AT=A?1
即
A T A ? 1 = E A^TA^{-1}=E ATA?1=E
即
[ a 1 T a 2 T … a n T ] ( a 1 , a 2 , … , a n ) = E \begin{bmatrix} a_1^T \\ a_2^T \\ \dots \\ a_n^T \end{bmatrix} (a_1,a_2,\dots,a_n)=E ?????a1T?a2T?…anT???????(a1?,a2?,…,an?)=E
也即
( a i T a j ) = ( δ i j ) (a_i^Ta_j) = (\delta_{ij}) (aiT?aj?)=(δij?)
相當于 n 2 n^2 n2個關系式
a i T a j = δ i j = { 1 when i = j 0 when i ≠ j a_i^Ta_j = \delta_{ij} = \begin{cases} 1 & \quad \text{when } i = j\\ 0 & \quad \text{when } i \neq j \end{cases} aiT?aj?=δij?={10?when i=jwhen i?=j?
因為 A T = A ? 1 A^T=A^{-1} AT=A?1,所以上述結論對 A A A的行向量亦成立，
（補充閱讀：同濟大學線性代數第五版第六章第三節）

5. Gram-Schmidt process for constructing an orthonoral basis 用格拉姆-施密特正交化構建正交基

Start with n n n linearly independent basis vectors v = { v 1 , v 2 , … , v n } v = \{ v_1,v_2,\dots,v_n \} v={v1?,v2?,…,vn?}. Then
e 1 = v 1 ∥ v 1 ∥ e_1 = \frac {v_1} {\|v_1\|} e1?=∥v1?∥v1??
u 2 = v 2 ? ( v 2 ? e 1 ) e 1 u_2 = v_2 - (v_2 \cdot e_1)e_1 u2?=v2??(v2??e1?)e1? so e 2 = u 2 ∥ u 2 ∥ e_2 = \frac {u_2} {\|u_2\|} e2?=∥u2?∥u2??
… and so on for u 3 u_3 u3? being the remnant part of v 3 v_3 v3? not composed of the preceding e e e-vectors, etc. …

6. Transformation in a Plane or other object

First transform into the basis referred to the reflection plane, or whicherev; E ? 1 E^{-1} E?1.
Then do the reflection or other transformation, in the plane of the object T E T_E TE?.
Then transform back intor the original basis E.
So our transformed vector r ′ = E T E E ? 1 r r' = ET_EE^{-1}r r′=ETE?E?1r

7. Eigenstuff 特征分解

To investigate the characteristics of the n n n by n n n matrix A A A, you are looking for the solutions the the equation,
A x = λ x Ax=\lambda x Ax=λx
where λ \lambda λ is a scalar eigenvalue. Eigenvalues will staisfy the following condition
( A ? λ I ) x = 0 (A-\lambda I)x = 0 (A?λI)x=0
where I I I is a n n n by n n n dimensional identity matrix

8. PageRank

To find the dominant eigenvector of link matrix L L L, the Power Method can be iteratively applied, staring from a uniform initial vector r ? \vec{r} r .
r i + 1 = L r i r^{i+1} = Lr^i ri+1=Lri
A damping factor, d, can be implement to stabilize this method as follows.
r i + 1 = d L r i + 1 ? d n r^{i+1} = dLr^i + \frac{1-d}{n} ri+1=dLri+n1?d?

第二部分 Multivariate 多元微積分

1. Defenition of a derivative 積分定義

f ′ ( s ) = d f ( x ) d x = lim ? x → 0 f ( x + Δ x ) ? f ( x ) Δ x f'(s) = \frac{\mathrm{d}f(x)}{\mathrm{d}x} = \lim\limits_{x \to 0} \frac{f(x + \Delta x) -f(x)}{\Delta x} f′(s)=dxdf(x)?=x→0lim?Δxf(x+Δx)?f(x)?

2. Time saving rules

• Sum Rule:
  d d x ( f ( x ) + g ( x ) ) = d d x ( f ( x ) ) + d d x ( g ( x ) ) \frac{\mathrm{d}}{\mathrm{d}x}(f(x)+g(x)) = \frac{\mathrm{d}}{\mathrm{d}x}(f(x)) + \frac{\mathrm {d}}{\mathrm{d}x}(g(x)) dxd?(f(x)+g(x))=dxd?(f(x))+dxd?(g(x))
• Power Rule:
  f ( x ) = a x b f(x) = ax^b f(x)=axb
  f ′ ( x ) = a b x b ? 1 f'(x) = abx^{b-1} f′(x)=abxb?1
• Product Rule:
  A ( x ) = f ( x ) g ( x ) A(x) = f(x)g(x) A(x)=f(x)g(x)
  A ′ ( x ) = f ′ ( x ) g ( x ) + f ( x ) g ′ ( x ) A'(x) = f'(x)g(x) + f(x)g'(x) A′(x)=f′(x)g(x)+f(x)g′(x)
• Chain Rule:
  If h = h ( p ) h = h(p) h=h(p) and p = p ( m ) p = p(m) p=p(m)
  then d h d m = d h d p × d p d m \frac{\mathrm {d}h}{\mathrm{d}m} = \frac{\mathrm{d}h}{\mathrm{d}p} × \frac{\mathrm{d}p}{\mathrm{d}m} dmdh?=dpdh?×dmdp?
• Total derivative:
  For the function f ( x , y , z , … ? ) f(x, y, z, \dots) f(x,y,z,…), where each variable is a function of parameter t t t, the total derivative is
  d f d t = ? f ? x d x d t + ? f ? y d y d t + ? f ? z d z d t + … \frac{\mathrm{d}f}{\mathrm{d}t} = \frac{\partial f}{\partial x}\frac{\mathrm{d}x}{\mathrm{d}t} + \frac{\partial f}{\partial y}\frac{\mathrm{d}y}{\mathrm{d}t} + \frac{\partial f}{\partial z}\frac{\mathrm{d}z}{\mathrm{d}t} + \dots dtdf?=?x?f?dtdx?+?y?f?dtdy?+?z?f?dtdz?+…

3. Derivatives of named functions 常見函式的導數

• ? ? x 1 x = ? 1 x 2 \frac{\partial}{\partial x}\frac{1}{\mathrm x}=-\frac{1}{\mathrm x^2} ?x??x1?=?x21?
• ? ? x sin ? x = cos ? x \frac{\partial}{\partial x}\sin x = \cos x ?x??sinx=cosx
• ? ? x cos ? x = ? sin ? x \frac{\partial}{\partial x}\cos x = - \sin x ?x??cosx=?sinx
• ? ? x exp ? x = exp ? x \frac{\partial}{\partial x}\exp x = \exp x ?x??expx=expx

4. Derivative structures

f = f ( x , y , z ) f = f(x,y,z) f=f(x,y,z)

• Jacobian:
  J f = [ ? f ? x , ? f ? y , ? f ? z ] \mathbf J_f = \begin{bmatrix} \frac{\partial f}{\partial x}, & \frac{\partial f}{\partial y}, & \frac{\partial f}{\partial z} \end{bmatrix} Jf?=[?x?f?,??y?f?,??z?f??]
• Hessian:
  H f = [ ? 2 f ? x 2 ? 2 f ? x ? y ? 2 f ? x ? z ? 2 f ? y ? x ? 2 f ? y 2 ? 2 f ? y ? z ? 2 f ? z ? x ? 2 f ? z ? y ? 2 f ? z 2 ] \mathbf H_f = \begin{bmatrix} \frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} & \frac{\partial^2 f}{\partial x \partial z} \\ \frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2} & \frac{\partial^2 f}{\partial y \partial z} \\ \frac{\partial^2 f}{\partial z \partial x} & \frac{\partial^2 f}{\partial z \partial y} & \frac{\partial^2 f}{\partial z^2} \end{bmatrix} Hf?=?????x2?2f??y?x?2f??z?x?2f???x?y?2f??y2?2f??z?y?2f???x?z?2f??y?z?2f??z2?2f??????

5. Taylor Series 泰勒展開式

5.1 Maclaurin 麥克勞林展開式

f ( x ) = f ( 0 ) + f ′ ( c ) ( x ) + 1 2 f ′ ′ ( 0 ) ( x ) 2 + ? = ∑ n = 0 ∞ f ( n ) ( 0 ) n ! ( x ) n f(x) = f(0) + f'(c)(x) + \frac{1}{2}f''(0)(x)^2 + \dots = \sum_{n=0}^{\infty}\frac{f^{(n)}(0)}{n!}(x)^n f(x)=f(0)+f′(c)(x)+21?f′′(0)(x)2+?=n=0∑∞?n!f(n)(0)?(x)n

5.2 泰勒展開式

• Univariate 一元:
  f ( x ) = f ( c ) + f ′ ( c ) ( x ? c ) + 1 2 f ′ ′ ( c ) ( x ? c ) 2 + ? = ∑ n = 0 ∞ f ( n ) ( c ) n ! ( x ? c ) n f(x) = f(c) + f'(c)(x-c) + \frac{1}{2}f''(c)(x-c)^2 + \dots = \sum_{n=0}^{\infty}\frac{f^{(n)}(c)}{n!}(x-c)^n f(x)=f(c)+f′(c)(x?c)+21?f′′(c)(x?c)2+?=n=0∑∞?n!f(n)(c)?(x?c)n

f ( x + Δ x ) = f ( x ) + f ′ ( x ) Δ x + 1 2 f ′ ′ ( x ) Δ x 2 = ∑ n = 0 ∞ f ( n ) ( x ) n ! Δ x n f(x+\Delta x) = f(x) + f'(x)\Delta x+ \frac{1}{2}f''(x)\Delta x^2 = \sum_{n=0}^{\infty}\frac{f^{(n)}(x)}{n!}\Delta x^{n} f(x+Δx)=f(x)+f′(x)Δx+21?f′′(x)Δx2=n=0∑∞?n!f(n)(x)?Δxn

• Multivariate 多元:
  f ( x ) = f ( c ) + J f ( c ) ( x ? c ) + … 1 2 ( x ? c ) t H f ( c ) ( x ? c ) + … f(x) = f(c) + \mathbf J_f(c)(x-c) + \dots \\ \frac{1}{2}(x-c)^t\mathbf H_f(c)(x-c) + \dots f(x)=f(c)+Jf?(c)(x?c)+…21?(x?c)tHf?(c)(x?c)+…
  x x x and c c c are vector for variable and constant

6. Optimization and Vector Calculus

• Newton-Raphson:
  x i + 1 = x i ? f ( x i ) f ′ ( x i ) x_{i+1} = x_i - \frac{f(x_i)}{f'(x_i)} xi+1?=xi??f′(xi?)f(xi?)?
• Grad:
  ? f = [ ? f ? x ? f ? y ? f ? z ] \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \\ \frac{\partial f}{\partial z} \end{bmatrix} ?f=?????x?f??y?f??z?f??????
• Directional Gradient:
  ? f ? r ^ \nabla f\cdot\hat{r} ?f?r^
• Gradient Descent:
  s n + 1 = s n ? γ ? f s_{n+1} = s_n - \gamma \nabla f sn+1?=sn??γ?f
• Lagrange Multipliers λ \lambda λ:
  ? f = λ ? g \nabla f = \lambda \nabla g ?f=λ?g
  [ ? f ? x ? f ? y ] = λ [ ? g ? x ? g ? y ] \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix} = \lambda \begin{bmatrix} \frac{\partial g}{\partial x} \\ \frac{\partial g}{\partial y} \end{bmatrix} [?x?f??y?f??]=λ[?x?g??y?g??]
  ? L ( x , y , λ ) = [ ? f ? x ? λ ? g ? x ? f ? y ? λ ? g ? y ? g ( x ) ] \nabla \mathcal{L} (x,y,\lambda) = \begin{bmatrix} \frac{\partial f}{\partial x} - \lambda \frac{\partial g}{\partial x} \\ \frac{\partial f}{\partial y} - \lambda \frac{\partial g}{\partial y} \\ -g(x) \end{bmatrix} ?L(x,y,λ)=????x?f??λ?x?g??y?f??λ?y?g??g(x)????
• Least Squares - χ 2 \chi^2 χ2 minimization:
  χ 2 = ∑ i n ( y i ? y ( x i ; a k ) ) 2 σ i \chi^2 = \sum_i^n \frac{(y_i-y(x_i;a_k))^2}{\sigma_i} χ2=i∑n?σi?(yi??y(xi?;ak?))2?
  criterion: ? χ 2 = 0 \nabla \chi^2 = 0 ?χ2=0
  a n e x t = a c u r ? γ ? χ 2 = a c u r + γ ∑ i n ( y i ? y ( x i ; a k ) ) σ i ? y ? a k a_{next} = a_{cur} - \gamma \nabla \chi^2 \\ = a_{cur} + \gamma \sum_i^n \frac{(y_i-y(x_i;a_k))}{\sigma_i} \frac{\partial y}{\partial a_k} anext?=acur??γ?χ2=acur?+γi∑n?σi?(yi??y(xi?;ak?))??ak??y?

第三部分 PCA (Principal Component Analysis) 主成分分析

1. 1-D datasets 一維資料

Given a data set D = { x 1 , ? ? , x N } D = \{x_1,\cdots,x_N\} D={x1?,?,xN?}, x n ∈ R x_n \in R xn?∈R,

• Mean Value
  E [ D ] = 1 N ∑ n = 1 N x n E[D]=\frac{1}{N}\sum_{n=1}^{N}x_n E[D]=N1?n=1∑N?xn?
• Variance
  V [ D ] = E [ ( x n ? μ ) 2 ] = 1 N ∑ n = 1 n ( x n ? μ ) 2 V[D]=E[(x_n-\mu)^2]=\frac{1}{N}\sum_{n=1}^{n}(x_n-\mu)^2 V[D]=E[(xn??μ)2]=N1?n=1∑n?(xn??μ)2

2. Definite symmetric matrix

Given a symmetric real matrix M ∈ R n × n M \in R^{n×n} M∈Rn×n, ? z ∈ R n × x + \forall z \in R^{n×x}+ ?z∈Rn×x+. When z T M z > 0 z^TMz > 0 zTMz>0, then M M M is a positive-definite matrix. When z T M z ≥ 0 z^TMz \geq 0 zTMz≥0, then M M M is a positiv semi-definite matrix.

3. higher-dimensional datasets 高維資料

Given a data set X = { x 1 , ? ? , x N } X = \{x_1,\cdots,x_N\} X={x1?,?,xN?}, x n ∈ R D × 1 x_n \in R^{D×1} xn?∈RD×1, X ∈ R D × N X \in R^{D×N} X∈RD×N
X = [ x 1 , 1 x 1 , 2 ? x 1 , N x 2 , 1 x 2 , 2 ? x 2 , N ? ? ? ? x D , 1 x D , 2 ? x D , N ] X = \begin{bmatrix} x_{1,1} & x_{1,2} & \cdots & x_{1,N} \\ x_{2,1} & x_{2,2} & \cdots & x_{2,N} \\ \vdots & \vdots & \ddots & \vdots \\ x_{D,1} & x_{D,2} & \cdots & x_{D,N} \end{bmatrix} X=??????x1,1?x2,1??xD,1??x1,2?x2,2??xD,2???????x1,N?x2,N??xD,N????????

• Mean Value
  μ = E [ X ] = 1 N ∑ n = 1 N x n = [ μ 1 μ 2 ? μ D ] ∈ R D × 1 \mu=E[X]=\frac{1}{N}\sum_{n=1}^{N}x_n = \begin{bmatrix} \mu_1 \\ \mu_2 \\ \vdots \\ \mu_D \end{bmatrix} \in R^{D×1} μ=E[X]=N1?n=1∑N?xn?=??????μ1?μ2??μD????????∈RD×1
• Variance
  V [ X ] = 1 N ∑ n = 1 N ( x n ? μ ) ( x n ? μ ) T = 1 N [ ( x n , 1 ? μ n , 1 ) ( x n , 1 ? μ ) T ( x n , 2 ? μ ) ( x n , 1 ? μ ) T ? ( x n , D ? μ ) ( x n , 1 ? μ ) T ( x n , 1 ? μ ) ( x n , 2 ? μ ) T ( x n , 2 ? μ ) ( x n , 2 ? μ ) T ? ( x n , D ? μ ) ( x n , 2 ? μ ) T ? ? ? ? ( x n , 1 ? μ ) ( x n , D ? μ ) T ( x n , 2 ? μ ) ( x n , D ? μ ) T ? ( x n , D ? μ ) ( x n , D ? μ ) T ] ∈ R D × D \begin{aligned}V[X]&=\frac{1} {N}\sum_{n=1}^N(x_n - \mu) (x_n - \mu)^T \\ &= \frac{1}{N} \begin{bmatrix} (x_{n,1} - \mu_{n,1})(x_{n,1} - \mu)^T & (x_{n,2} - \mu)(x_{n,1} - \mu)^T & \cdots & (x_{n,D} - \mu)(x_{n,1} - \mu)^T \\ (x_{n,1} - \mu)(x_{n,2} - \mu)^T & (x_{n,2} - \mu)(x_{n,2} - \mu)^T & \cdots & (x_{n,D} - \mu)(x_{n,2} - \mu)^T \\ \vdots & \vdots & \ddots & \vdots \\ (x_{n,1} - \mu)(x_{n,D} - \mu)^T & (x_{n,2} - \mu)(x_{n,D} - \mu)^T & \cdots & (x_{n,D} - \mu)(x_{n,D} - \mu)^T \end{bmatrix} \in R^{D×D} \end{aligned} V[X]?=N1?n=1∑N?(xn??μ)(xn??μ)T=N1???????(xn,1??μn,1?)(xn,1??μ)T(xn,1??μ)(xn,2??μ)T?(xn,1??μ)(xn,D??μ)T?(xn,2??μ)(xn,1??μ)T(xn,2??μ)(xn,2??μ)T?(xn,2??μ)(xn,D??μ)T??????(xn,D??μ)(xn,1??μ)T(xn,D??μ)(xn,2??μ)T?(xn,D??μ)(xn,D??μ)T???????∈RD×D?
  D D D維資料的方差為 D × D D \times D D×D的矩陣，對角線上的元素 a i i a_{ii} aii?為第 i i i維資料第方差，其它元素 a i j a_{ij} aij?是第 i i i維與 j j j維資料第協方差，

4. Effect of Linear Transformations 線性變換對均值與方差對影響

Given a data set D = { x 1 , ? ? , x N } D = \{x_1,\cdots,x_N\} D={x1?,?,xN?}, x n ∈ R D × 1 x_n \in R^{D×1} xn?∈RD×1, D ∈ R D × N D \in R^{D×N} D∈RD×N, with
E [ D ] = μ E[D] = \mu E[D]=μ
V [ D ] = Q V[D] = Q V[D]=Q
linear transformations:
x i ′ = A x i + b x'_i = Ax_i +b xi′?=Axi?+b
then
E [ D ′ ] = A μ + b E[D'] = A\mu+b E[D′]=Aμ+b
V [ D ′ ] = A Q A T V[D'] = AQA^T V[D′]=AQAT
where D ′ = { x 1 ′ , x 2 ′ , ? ? , x N ′ } D' = \{x'_1,x'_2,\cdots,x'_N\} D′={x1′?,x2′?,?,xN′?}

5. Dot product 點積

• dot product
  x T y = ∑ d = 1 D x d y d , x , y ∈ R D x^Ty=\sum_{d=1}^Dx_dy_d, \quad x,y \in R^D xTy=d=1∑D?xd?yd?,x,y∈RD
• length
  ∥ x ∥ = x T x \|x\|=\sqrt{x^Tx} ∥x∥=xTx ?
• angle ω \omega ω between vectors x x x, y y y
  c o s ω = x T y ∥ x ∥ ∥ y ∥ cos\omega=\frac{x^Ty}{\|x\|\|y\|} cosω=∥x∥∥y∥xTy?

6. Inner product 內積

Consider a vector space V. A positive definete, symmetric bilinear mapping ? ? , ? ? : V × V → R \langle\cdot,\cdot\rangle: V \times{} V \to R ??,??:V×V→R is called an inner product on V V V.

• symmetric: ? x , y ∈ V ? x , y ? = ? y , x ? \forall x, y \in V \quad \langle x,y \rangle = \langle y,x\rangle ?x,y∈V?x,y?=?y,x?
• positive definite: ? x ∈ V \ { 0 } ? x , x ? > 0 , ? 0 , 0 ? = 0 \forall x \in V\backslash\{0\} \quad \langle x, x \rangle > 0, \langle 0,0 \rangle=0 ?x∈V\{0}?x,x?>0,?0,0?=0
• bilinear: ? x , y , z ∈ V , λ ∈ R \forall x,y,z \in V, \lambda \in R ?x,y,z∈V,λ∈R
  ? λ x + y , z ? = λ ? x , z ? + ? y , z ? \langle \lambda x+y,z\rangle = \lambda\langle x,z\rangle+\langle y,z\rangle ?λx+y,z?=λ?x,z?+?y,z?
  ? x , λ y + z ? = λ ? x , y ? + ? x , z ? \langle x,\lambda y+z\rangle = \lambda\langle x,y\rangle+\langle x,z\rangle ?x,λy+z?=λ?x,y?+?x,z?
• length of a vector x ∈ V x \in V x∈V
  ∥ x ∥ = ? x , x ? \|x\|=\sqrt{\langle x,x\rangle} ∥x∥=?x,x? ?
• distance between two vectors x , y ∈ V x,y \in V x,y∈V
  d ( x , y ) = ∥ x ? y ∥ = ? x ? y , x ? y ? d(x,y)=\|x-y\|=\sqrt{\langle x-y,x-y\rangle} d(x,y)=∥x?y∥=?x?y,x?y? ?
• angle ω \omega ω between two vectors x , y ∈ V x,y\in V x,y∈V
  c o s ω = ? x , y ? ∥ x ∥ ∥ y ∥ cos\omega=\frac{\langle x,y\rangle}{\|x\|\|y\|} cosω=∥x∥∥y∥?x,y??
  where ∥ x ∥ \|x\| ∥x∥ is defined via ineer product as ? x , x ? \sqrt{\langle x,x\rangle} ?x,x? ?

點積向其它資料型別對拓展

• Ineer product for continuous data

? f , g ? = ∫ a b f ( x ) g ( x ) d x \langle f,g \rangle =\int\limits_a^b f(x)g(x)\mathrm{d}x ?f,g?=a∫b?f(x)g(x)dx

• Inner product for random variables

? X , Y ? = C o v ( X , Y ) \langle X,Y \rangle=Cov(X,Y) ?X,Y?=Cov(X,Y)

7. Projection 投影

7.1 Projection onto 1D subspaces

Consider a vector space V V V and a subspace U U U of V V V. With a basis vector b b b of U U U, we obtain the orthogonal projection of any x ∈ V x \in V x∈V onto U U U via
π u ( x ) = λ b , λ = b T x b T b = b T x ∥ b ∥ 2 \pi_u(x) = \lambda b, \quad \lambda=\frac{b^Tx}{b^Tb}=\frac{b^Tx}{\|b\|^2} πu?(x)=λb,λ=bTbbTx?=∥b∥2bTx?
where λ \lambda λ is the coordinate of π u ( x ) \pi_u(x) πu?(x) with respect to b b b.
The projection matrix P P P is
P = b b T b T b = b b T ∥ b ∥ 2 P=\frac{bb^T}{b^Tb}=\frac{bb^T}{\|b\|^2} P=bTbbbT?=∥b∥2bbT?
such that
π u ( x ) = P x \pi_u(x)=Px πu?(x)=Px
for all x ∈ V x\in V x∈V

7.2 Projection onto k k k-dimensional subspaces

Consider a vector space V V V and a subspace U U U of V V V. With a basis vector b 1 , ? ? , b k b_1,\cdots,b_k b1?,?,bk? of U U U, we obtain the orthogonal projection of any x ∈ V x \in V x∈V onto U U U via
π u ( x ) = B λ , λ = ( B T B ) ? 1 B T x \pi_u(x) = B\lambda,\quad \lambda=(B^TB)^{-1}B^Tx πu?(x)=Bλ,λ=(BTB)?1BTx
B = ( b 1 ∣ ? ∣ b k ) ∈ R n × k B=(b_1|\cdots|b_k)\in R^{n\times k} B=(b1?∣?∣bk?)∈Rn×k
where λ \lambda λ is the coordinate of π u ( x ) \pi_u(x) πu?(x) with respect to b 1 , ? ? , b k b_1,\cdots,b_k b1?,?,bk? of U U U.
The projection matrix P P P is
P = B ( B T B ) ? 1 B T P=B(B^TB)^{-1}B^T P=B(BTB)?1BT
such that
π u ( x ) = P x \pi_u(x)=Px πu?(x)=Px
for all x ∈ V x\in V x∈V

8. PCA derivation 主成分分析推導

8.1 Setting up ( X n = ∑ i = 1 D β i n b i X_n=\sum_{i=1}^D\beta_{in}b_i Xn?=∑i=1D?βin?bi?, X n ~ = ∑ i = i M β i n b i \tilde{X_n} = \sum_{i=i}^M\beta_{in}b_i Xn?~?=∑i=iM?βin?bi?, J = 1 N ∥ X n ? X n ~ ∥ 2 \mathbf{J} =\frac{1}{N}\|X_n-\tilde{X_n}\|^2 J=N1?∥Xn??Xn?~?∥2, S = 1 N ∑ n = 1 N X n X n T \mathrm{S}=\frac{1}{N}\sum_{n=1}^N X_nX_n^T S=N1?∑n=1N?Xn?XnT?)

Given a data set X = x 1 , ? ? , x n X={x_1,\cdots,x_n} X=x1?,?,xn?, x i ∈ R D x_i\in R^D xi?∈RD, E [ X n ] = 0 \mathrm{E}[X_n]=0 E[Xn?]=0, Original basis A = ( a 1 , ? ? , a n ) A=(a_1,\cdots,a_n) A=(a1?,?,an?), project to a new orthonormal basis B = ( b 1 , ? ? , b n ) , b n ∈ R D B=(b_1,\cdots,b_n),\quad b_n \in R^D B=(b1?,?,bn?),bn?∈RD

S = 1 N ∑ n = 1 N ( x n ? μ ) ( x n ? μ ) T = 1 N ∑ n = 1 N X n X n T ( E [ X n ] = 0 ) \begin{aligned}\mathrm{S} &= \frac{1} {N}\sum_{n=1}^N (x_n - \mu) (x_n - \mu)^T \\ &= \frac{1}{N}\sum_{n=1}^N X_{n} X_{n}^T \quad (\mathrm{E}[X_n]=0) \end{aligned} S?=N1?n=1∑N?(xn??μ)(xn??μ)T=N1?n=1∑N?Xn?XnT?(E[Xn?]=0)?
X X X represented in new basis
X n = ∑ i = 1 D β i n b i X_n=\sum_{i=1}^D\beta_{in}b_i Xn?=i=1∑D?βin?bi?
Our goal is represent X n X_n Xn? in D-dimentional space to a lower M-dimentional
X n ~ = ∑ i = i M β i n b i \tilde{X_n} = \sum_{i=i}^M\beta_{in}b_i Xn?~?=i=i∑M?βin?bi?
with minimum difference between X n X_n Xn? and X n ~ \tilde{X_n} Xn?~?. The cost function
J = 1 N ∥ X n ? X n ~ ∥ 2 \mathbf{J} =\frac{1}{N}\|X_n-\tilde{X_n}\|^2 J=N1?∥Xn??Xn?~?∥2

8.2 got coordinate/code β i n \beta_{in} βin? ( β i n = X n T b i \beta_{in}=X_n^Tb_i βin?=XnT?bi?)

? J ? β i n = ? J ? X n ~ ? X n ~ ? β i n = ? 2 N ( X n ? X n ~ ) T b i = ? 2 N ( X n ? ∑ i = i M β i n b i ) T b i ( X n ~ = ∑ i = i M β i n b i ) = ? 2 N ( X n ? β i n ∑ i = i M b i ) T b i ( β i n is scalar ) = ? 2 N ( X n T b i ? β i n b i T b i ) ( ONB ) = ? 2 N ( X n T b i ? β i n ) ( ONB ) \begin{aligned} \frac{\partial\mathbf{J}}{\partial\beta_{in}} &= \frac{\partial\mathbf{J}}{\partial\tilde{X_n}}\frac{\partial\tilde{X_n}}{\partial\beta_{in}} \\ &= -\frac{2}{N}(X_n-\tilde{X_n})^Tb_i \\ &= -\frac{2}{N}(X_n-\sum_{i=i}^M\beta_{in}b_i)^Tb_i \quad (\tilde{X_n} = \sum_{i=i}^M\beta_{in}b_i) \\ &= -\frac{2}{N}(X_n-\beta_{in}\sum_{i=i}^Mb_i)^Tb_i \quad (\beta_{in}\text{ is scalar})\\ &= -\frac{2}{N}(X_n^Tb_i-\beta_{in}b_i^Tb_i) \quad (\text{ONB}) \\ &= -\frac{2}{N}(X_n^Tb_i-\beta_{in}) \quad (\text{ONB}) \end{aligned}

轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/208946.html

標籤：其他

上一篇：周期信號的傅里葉級數展開分析（利用MATLAB）

下一篇：【數學建模】1層次分析法模型部分