Mini-Batch Gradient Descent

Mini-Batch Gradient Descent

1. What is Mini-Batch Gradient Descent?

Mini-Batch Gradient Descent is an algorithm between the Batch Gradient Descent and Stochastic Gradient Descent. Concretly, this use some(not one or all) examples(M) for each iteration.

2. Compute Effort

The compute time of this algorithm depends on the examples. It not stable, but the worst case is like Batch Gradient Descent: O(N2)

The table below shows the different among these there Gradient Descent

Batch Gradient Descent	Mini-Batch Gradient Descent	Stochastic Gradient Descent
use 1 example in each iteration	use some examples	use all example in each iteration
relative compute loose	somewhat in between	relative compute intensive

3. Gradient Descent Formula

For all θi

∂Jθ∂θi=1m∑i=1M[hθ(xi)−yi]⋅(xi)

E.g.,

two parameters θ0,θ1 –> hθ(x)=θ0+θ1x1

For i = 0 :

∂Jθ∂θ0=1m∑i=1M[hθ(xi)−yi]⋅(x0)

For i = 1:

∂Jθ∂θ1=1m∑i=1M[hθ(xi)−yi]⋅(x1)

Note that the datasets need to be shuffled before iteration.