正定与最小二乘法的联系

此博客停止更新迁移至SnailDove’s Blog，查看本文点击此处

positive definite

ATA

least squares approximations

positive definite

When a symmetric matrix AA has one of these five properties, it has them all and AA is positive definite:

all n eigenvalue are positive.

all n principal minors(n upper left determinants) are positive.

all n pivots are positive.

xTAxxTAx is positive except when x=0x=0 (this is usually the definition of positive definiteness and the energy-based definition).

AA equals RTRRTR for a matrix RR with independent columns.

Let us prove the fifth rule. If A=RTRA=RTR, then

xTAx====≥xTRTRx(xTRT)Rx(Rx)TRx∥Rx∥0xTAx=xTRTRx=(xTRT)Rx=(Rx)TRx=‖Rx‖≥0

And the columns of RR are also independent, so ∥Rx∥=xTAx>0‖Rx‖=xTAx>0, except when xx=0 and thus AA is positive definite.

ATAATA

Am×nAm×n is almost certainly not symmetric, but ATAATA is square (m by m) and symmetric. We can easily get the following equations through left multiplying ATAATA by xTxT and right multiplying ATAATA by xx:

xTATAx====≥xT(ATA)x(xTAT)Ax(Ax)T(Ax)∥Ax∥0xTATAx=xT(ATA)x=(xTAT)Ax=(Ax)T(Ax)=‖Ax‖≥0

If Am×nAm×n has rank nn (independent columns), then except when x=0x=0, Ax=∥Ax∥=xT(ATA)x>0Ax=‖Ax‖=xT(ATA)x>0 and thus ATAATA is positive definite. And vice versus.

Besides, ATAATA is invertible only if Am×nAm×n has rank nn (independent columns). To prove this, we assume Ax=0Ax=0, then:

Ax(Ax)T(Ax)(xTAT)(Ax)xTAT(Ax)(ATA)x=====000xT00Ax=0(Ax)T(Ax)=0(xTAT)(Ax)=0xTAT(Ax)=xT0(ATA)x=0

From the above equations, we know solutions of Ax=0Ax=0 are also solutions of (ATA)x=0(ATA)x=0. Because Am×nAm×n has a full set of column rank (independent columns), Ax=0Ax=0 only has a zero solution as well as (ATA)x=0(ATA)x=0. Moreover, if ATAATA is invertible, then Am×nAm×n has rank nn (independent columns). We also notice that if AA is square and invertible, then ATAATA is invertible.

Overall, if all columns of Am×nAm×n are mutual independent, then (ATA)(ATA) is invertible and positive definite as well, and vice versus.

least squares approximations

We have learned that least squares approximations comes from projection :

b−p=e⇒AT(b−Ax^)=0⇒ATAx^=ATbb−p=e⇒AT(b−Ax^)=0⇒ATAx^=ATb

Consequently, only if ATAATA is invertible, then we can use linear regression to find approximate solutions x^=(ATA)−1ATbx^=(ATA)−1ATb to unsolvable systems of linear equations.

According to the reasonibefore, we know as long as all columns of Am×nAm×n are mutual independent, then ATAATA is invertible. At the same time we ought to notice that the columns of Am×nAm×n are guaranteed to be independent if they are orthoganal and even orthonormal.

In another prospective, if ATAATA is positive definite, then Am×nAm×n has rank nn (independent columns) and thus ATAATA is invertible.

Overall, if ATAATA is positive definite or invertible, then we can find approximate solutions of least square.