机器学习实践指南（五）—— GD/SGD/MSGD 伪代码演示

GD：梯度下降

while True:
loss = f(params)
d_loss_wrt_params = ...
params -= eta * d_loss_wrt_params
if <stopping condition met>:
return params

SGD：随机梯度下降

for x_i, y_i in training_data:
loss = f(params, x_i, y_i)
d_loss_wrt_params = ...
params -= eta * d_loss_wrt_params
if <stopping condition met>:
return params

for j in range(epochs):
random.shuffle(training_data)
for x_i, y_i in training_data:
...

MSGD（Minibatch SGD）：块随机梯度下降

n = len(training_data)
mini_batch_size = ...
mini_batches = [training_data[k:k+mini_batch_size] for k in range(0, n, mini_batch_size)]
for mini_batch in mini_batches:
loss = f(params, mini_batch)
d_loss_wrt_params = ...
params -= eta * d_loss_wrt_params
if <stopping condition met>：
return params