Nesterov's method

Problem formulation[1]:

minimizef(x),x∈Rn

Nesterovs method generates a sequence of search points xi,i=0,1,..., and the sequence of approximate solutions yi,i=1,2,..., along with auxiliary sequences of vectors qi, positive reals Li abd reals ti≥1, according to the following rules:

Initialization: Set

x0=y1=q0=0;t0=1;

choose as L0 an arbitrary positive initial estimate of L(f).

i-th step, i≥1: Given xi−1,yi,qi−1,Li−1,ti−1, act as follows:

set

xi=yi−1ti−1(yi+qi−1)

and compute f(xi),f′(xi);

Testing sequentially the values l=2jLi−1,j=0,1,..., find the first value of l such that

f(xi−1lf′(xi))≤f(xi)−12l|f′(xi)|2

set Li equal to the resulting value of l.

Set

yi+1=xi−1Lif′(xi),qi=qi−1+ti−1Lif′(xi),

define ti as the larger root of the equation

t2−t=t2i−1

and loop.

Convergence rate: O(N−2).

[1] Y. Nesterov. Introductory Lectures on Convex Optimization. Kluwer Academic Publishers, 2003.