Brief overview of backward and forward

Let’s say we only feed in one data point.

out = model:forward(xi) computes fw(xi) where fw is our model with its current parameters w, and stores the result in out.

loss = criterion:forward(out, yi) computes the loss ℓ(fw(xi),yi) with respect to the true value yi.

dl_dout = criterion:backward(out, yi) computes ∂ℓ(...)∂fw(xi).

model:backward(xi, dl_dout) computes ∂ℓ(...)∂w and stores this gradient in a place we have a reference to, usually called gradParameters in our code.