Lecture2-3Guarantee of PLA

Linear Separable

If PLA halts(no mistakes),(necessary condition)  allows some w to make no mistake

Call  linear separable

Linear Separable ⟺ exists perfect wf such that yn=sign(wTfxn)

<
24000
span class="mrow" id="MathJax-Span-7240">wf perfect hence every xn correctly away from the line:

yn(t)wTfxn(t)≥minnynwTfxn>0

Fact:wt gets closer to wf by updating.

wTfwt+1=wTf(wt+yn(t)xn(t))≥wTfwt+yn(t)wTfxn(t)≥wTfwt+minnynwTfxn>wTfwt

Fact: wt doesn’t grow too fast.

Derivation

After T derivations, we can have:

Remark1

∵wTfwT∴wTfwT∥wTf∥=wTf(wT−1+yn(T)xn(T))=wTfwT−1∥wTf∥+wTfyn(T)xn(T)∥wTf∥≥wTfwT−1∥wTf∥+minnwTfynxn∥wTf∥≥wTfwT−1∥wTf∥+ρ≥wTfwT−2∥wTf∥+2ρ...≥Tρ

Remark2

∥wT∥2=∥wT−1+yn(T)xn(T)∥2=∥wT−1∥2+∥yn(T)xn(T)∥2+2wTT−1yn(T)xn(T)≤∥wT−1∥2+∥xn(T)∥2≤∥wT−1∥2+maxn∥xn∥2≤∥wT−1∥2+R2≤∥wT−2∥2+2R2...≤TR2

∴ the perceptron iterates at most T iterations before it stops, if  is linear separable

∴1≥wTfwT∥wf∥∥wT∥≥TρT‾‾√R=T‾‾√ρR∴T≤R2ρ2

where ρ=minnwTfynxn∥wTf∥ and R=maxn∥xn∥.