MIT 线性代数

vedio 9

Suppose A m by n while m < n, then there are nonzero solutions to AX = 0.

Reason: There will be free variables. (rank < n)

When v1,…,vnv_1,…, v_n are columns of A, they are independent if nullspace of A is only {zero vector}. (rank = n)

They are dependent if AC = 0 for some nonzero C.

Vectors v1,...,vlv_1,...,v_l span a space means: The space consists all combinations of those vectors.

Basis for a space is a sequence of vectors v1,...,vdv_1,..., v_d with 2 properties

1 They are independent

2 They span the space

Every basis for the space has the same number of vectors.

rank(A) = number of pivot columns = dimention of C(A)

dim N(A) = number of free variables = n - r

vedio 10

4 foundermental sub-spaces:

columnspace: C(A) in RmR^m, to be exactly, it is r, and r < m

nullspace: N(A) in RnR^n, it is (n - r)

rowspace = all colums of row = all columns of ATA^T: C(AT)C(A^T) in RnR^n

nullspace of ATA^T: N(AT)N(A^T) in RmR^m= left null space of A

left null space 名字由来:

ATy=0⇒(ATy)T=0⇒yTA=0A^Ty = 0 \Rightarrow (A^Ty)^T = 0 \Rightarrow y^TA = 0

Ex:

A = [1, 1, 1, 1]

dim rowspace of A = 1

dim N(A) = n - r = 3

dim C(A) = 1

dim N(ATA^T) = 0

C(A)	N(A)
pivot columns	special columns
r	n - r

行变换以后 same row space, different column space

vedio 11

矩阵空间R3X3R^{3X3}是9维，basis有9个，分别如下：

⎡⎣⎢100000000⎤⎦⎥
\begin{bmatrix}
1 & 0 & 0 \\
0 & 0 & 0 \\
0 & 0 & 0 \\
\end{bmatrix}

⎡⎣⎢000100000⎤⎦⎥
\begin{bmatrix}
0 & 1 & 0 \\
0 & 0 & 0 \\
0 & 0 & 0 \\
\end{bmatrix}

⎡⎣⎢000000100⎤⎦⎥
\begin{bmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
0 & 0 & 0 \\
\end{bmatrix}

……

⎡⎣⎢000000001⎤⎦⎥
\begin{bmatrix}
0 & 0 & 0 \\
0 & 0 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}

S是三维对称矩阵，U是上三角矩阵(似乎基的数量等于维数)

dim S = 6, dim U = 6

dim(S ∩\cap U) = 3

dim(S + U) = 9 (S + U can span R3R^3)

Graph = {nodes, edges}

vedio 14

number	name	dim
1	row space	r
2	null space	n - r
3	column space	r
4	null space of ATA^T	m - r

1 is orthogonal to 2, 3 is orthogonal to 4

if x is orthogonal to y, xTy=0x^Ty = 0, then xTx+yTy=(x+y)T(x+y)x^Tx + y^Ty = (x + y)^T(x + y)

Proof:

xTx+yTy=xTx+xTy+yTx+yTyx^Tx + y^Ty = x^Tx + x^Ty + y^Tx + y^Ty

⇔0=xTy+yTx=2xTy\Leftrightarrow 0 = x^Ty + y^Tx = 2x^Ty

⇔0=xTy\Leftrightarrow 0 = x^Ty

Subspace S is orthogonal to Subspace T means: every vector in S is orthogonal to every vector in T.

Nullspace(n - r) and rowspace(r) are orthogonal Complements(补) in Rn.R^n.

Nullspace contains all vectors ⊥\bot rowspace.

Ax = b(m > n), when there is no solution: ATAx^=ATbA^TA\hat x = A^Tb

N(ATA)=N(A)N(A^TA) = N(A)

rank of ATA=rank of Arank\ of\ A^TA = rank\ of\ A

ATA is invertible exactly if A has independent columns.A^TA\ is\ invertible\ exactly\ if\ A\ has\ independent\ columns.

vedio 15

有关投影，b在a上的投影为p，p = ax，e = b - p，

a 和e正交，因此aTe=0a^Te = 0

⇔aT(b−p)=0\Leftrightarrow a^T(b - p) = 0

⇔aT(b−ax)=0\Leftrightarrow a^T(b - ax) = 0

⇔aTax=aTb\Leftrightarrow a^Tax =a^Tb

⇔x=aTbaTa\Leftrightarrow x = \frac {a^Tb}{a^Ta}

p=ax=aaTbaTa=aaTaTabp = ax = a\frac {a^Tb}{a^Ta} = \frac{aa^T}{a^Ta} b

project matric(投影矩阵为) P=aaTaTa\mathbb P = \frac{aa^T}{a^Ta}

PT=P\mathbb P^T = \mathbb P 从上式就可以看出来

P2=P\mathbb P^2 = \mathbb P 投影两次和一次的结果是一样的

为什么要投影，why project?

Because Ax = b may have no solution, then solve Ax^=PA\hat x =\mathbb P instead

二维情况下:

P=Ax^， find x^\mathbb P = A\hat x ，\ find \ \hat x

key： e=b−Ax^e = b - A\hat x is perpendicular to plane

aT1(b−Ax^)=0a_1^T(b - A\hat x) = 0

aT2(b−Ax^)=0a_2^T(b - A\hat x) = 0

[aT1aT2](b−Ax^)=[00]
\left[
\begin{matrix}
a_1^T \\
a_2^T \\
\end{matrix}
\right]
(b - A\hat x) =
\left[
\begin{matrix}
0 \\
0 \\
\end{matrix}
\right]

⇔AT(b−Ax^)=0\Leftrightarrow A^T(b - A\hat x) = 0

⇔ATAx^=ATb\Leftrightarrow A^TA\hat x = A^Tb

⇔x^=ATbATA\Leftrightarrow \hat x = \frac {A^Tb}{A^TA}

p=Ax^=A(ATA)−1ATbp = A\hat x = A(A^TA)^{-1}A^Tb

投影矩阵P=A(ATA)−1AT\mathbb P = A(A^TA)^{-1}A^T

PT=P, P2=P\mathbb P^T = \mathbb P,\ \mathbb P^2 = \mathbb P

vedio 16

P=A(ATA)−1AT\mathbb P = A(A^TA)^{-1}A^T

If b in column space Pb=b\mathbb Pb = b

If b ⊥\bot column space Pb=0\mathbb Pb = 0

数学上的解释:

Column space 与 Nullspace of ATA^T 正交，

如果b in column space, b=Ax,b = Ax,

Pb=A(ATA)−1ATb=A(ATA)−1ATAx=Ax=b \mathbb Pb = A(A^TA)^{-1}A^Tb = A(A^TA)^{-1}A^TAx = Ax = b

如果b in Null space of AT,ATb=0⇒Pb=0A^T, A^Tb = 0 \Rightarrow \mathbb Pb = 0

对于任意向量b，在C(A)上的投影为p，在N(A)上的投影为e， p + e = b

其中p=Pb,e=(I−P)bp = \mathbb Pb, e = (I - \mathbb P)b

也很好理解，e=b−p=b−Pb=(I−P)be = b - p = b - \mathbb Pb = (I - \mathbb P)b

线性规划的本质就是 Minimize ||Ax−b||2=e2||Ax - b||^2 = e^2

当我们没办法求出Ax = b时，只能求Ax^=pA\hat x = p, p是b在C(A)上的投影

Ax^=p=Pb=A(ATA)−1ATbA\hat x = p =\mathbb Pb = A(A^TA)^{-1}A^Tb

⇔ATAx^=ATb\Leftrightarrow A^TA\hat x = A^Tb

e=b−p=b−Ax^e = b - p = b - A\hat x

If A has independent Columns space, then ATAA^TA is invertible.