机器学习基石 - Theory of Generalization

机器学习基石上 (Machine Learning Foundations)—Mathematical Foundations

Hsuan-Tien Lin, 林轩田，副教授 (Associate Professor)，资讯工程学系 (Computer Science and Information Engineering)

Theory of Generalization

Restriction of Break Points

growth function mH(N)mH(N): max number of dichotomies

漏出一线曙光的点 break point

break point kk restricts maximum possible mH(N)mH(N) a lot for N>kN>k

Bounding Function: Basic Cases

B(N,k)B(N,k): maximum possible mH(N)mH(N) when break point = k



表格

Bounding Function: Inductive Cases

B(4,3)B(4,3) 的估计





Putting It All Together



B(N,k)≤∑k−1i=0(Ni)B(N,k)≤∑i=0k−1(Ni)

数学归纳法

C
12243
iN−1+C i+1N−1=C i+1NCN−1 i+CN−1 i+1=CN i+1

actually ≤≤ can be ==

B(N,k)≥2B(N−1,k−1)+(B(N−1,k)−B(N−1,k−1))B(N,k)≥2B(N−1,k−1)+(B(N−1,k)−B(N−1,k−1))

can bound mH(N)mH(N) by only one break point

A Pictorial Proof

Step 1: Replace EoutEout by E ′inEin ′



Step 2: Decompose HH by Kind



Step 3: Use Hoeffding without Replacement



Vapnik-Chervonenkis (VC) bound