CS231n课程作业(一) SVM classifier

一、理论知识

1. score function

将原始数据映射到每一类上计算得分的函数。

(PS: CNN也是将原始输入的像素映射成类目得分，只不过其中间映射更加复杂，参数更多。)

对于SVM，score function为：



其中，xi指第i张图片，W为权重(可以理解为图像上每个像素点都有weight)，b为偏移量bias。

这里可以看出，优化时不仅要调整W，还需调整b。为了简化优化，将b并入到W中，形成一个新的W，这样优化时只需对W优化即可。处理方法如下：



因此，新的score function为：

2. loss function

用于描述模型预测值与真实值的不一致程度。

loss function is made of two component: data loss and regularization loss.

data loss为：



其中，Li表示第i张图片的data loss， sj指第i张图片对第j类的score，syi指第i张图片所属正确类的score。这里的1其实是一个给定的delta值，也可以设为其他常数，delta的含义是正确类的score不仅理应比其他类的score高，而且应该高出delta，否则将产生data loss。

训练模型的目的就是让loss值最小，即optimization。

Optimization： start with a random W and find a W that minimizes the loss.

但是，有时多组W可以得到相同的data loss值，比如：



这样W就是不确定的。因此，需要通过正则化选取一组最合适的W。

Regularization loss :(in common use: L2 regularization)



因此，完整的loss function为：



其中，N为训练样本的数量，lambda为正则项系数。lambda由cross-validation确定。(还有learning rate)

3. 梯度相关知识

梯度：指loss function对W求导，这样可以得到函数下降最快的方向。

numerical gradient: 即从求导的定义出发



analytic gradient: 直接用Li对W求偏导

numerical gradient is easy to write but slow and approximate, and analytic gradient is fast and exact but error-prone. In practice, derive analytic gradient, then check your implementation with numerical gradient.

这是data loss 的展开形式：



其中，Li指第i张图片的data loss值，Wj指W矩阵的第j列(即权重矩阵对第j类的权)，Wyi指W矩阵的第yi列(权重矩阵对第i张图片的正确所属类的权)。本作业中：W:(3073,10), X:(N,3073), y:(N,)

这里变量是W，求analytic gradient：

(更新：下面为错解，正解见下面的更新)



———————————————————————2018-1-10更新——————————————————————



————————————————————————–更新完毕———————————————————————–

*注：这两种梯度求法在编写svm classifier中很关键！

梯度下降： 梯度下降即通过反复的迭代运算，计算gradient进行权值更新，最后停留在loss function的低值点。(此时的W是使loss function最低的W。)

作一个直观理解：比如你半夜小便，眼睛睁不开，不开灯，对着马桶开始飙尿。你的目的是尽快(因为尿量总共就那么多)尿到马桶中有水的区域。所以你开始randomly尿到一侧马桶内壁，然后不断的下降落尿点。这里你需要get到最快的下降方向，也就是gradient，沿着这个梯度负方向下降一定的步长(step_size或learning rate)，一点一点的尿到马桶的最低点。此时对卫生间的环境损失最小(minimize the loss function)，然后你按下冲水按钮，就可以安心睡觉了。

4. 迭代相关知识

batch_size: number of training examples to use at each iteration.

迭代的具体工作：

(1) 取batch_size个样本，evaluate the loss & gradient.

(2) update W: W += -learning_rate * gradient

(3) 利用更新好的W，重复(1)、(2)，直到迭代结束。

Mini-batch Gradient Descent: In practice, only use a small portion of the training set to compute the gradient of loss function.

缺点：得出的gradient很有可能是一个噪声

优点：使用小样本计算可以提高运算速度，可以计算更多的次数来evaluate gradient，从而得出最低的loss function。或者也可以降低运算循环次数，使用更加精确的gradient。另外，在实际中使用所有数据计算梯度并不可行。(GPU hold不住)

learning_rate: 每一次迭代中W沿梯度负方向移动的大小(见上面“迭代的具体工作”)，lr决定了参数移动到最优值的速度快慢。lr的值由cross-validation确定。(还有正则项系数lambda也是)

if very high: loss function不会收敛，会越来越大。

if high: loss function会收敛，但可能收敛后的函数值会越过最优值，卡在一个比较高的位置，即没有找到最低的loss function。由于更新速度快，你不能确定要优化的函数是否进入一个局部最小值，你以为局部最小值是最小值，但实际并不是。

if low: 更新速度慢，收敛需要很长时间。

if good: then good.

策略：先set a high lr，再一点点降低。

二、SVM classifier

1. 数据介绍

50000张图片，32x32x3， 10类。其中training set为49000张， test set为1000张。

从training set中产生validation set 1000张和development set 500张。

X_train: (49000,3072) , y_train: (49000,)

X_val: (1000,3072) , y_val: (1000,)

X_test: (1000,3072) , y_test: (1000,)

X_dev: (500,3072) , y_dev: (500,)

*以上3072在运算中均增加为3073，原因不解释(除非你没好好看上面的理论部分)

W: (3073,10)

dW: (3073,10)

2. 预处理: 去均值

这样的处理其实很常见，例如摄影测量里的重心化。

first: compute the image mean based on the training data

second: subtract the mean image from train and test data

third: append the bias dimension of ones (i.e. bias trick) so that our SVM only has to worry about optimizing a single weight matrix W.

3. 计算loss和梯度

这里的梯度是analytic gradient

方法一 naive版：

def svm_loss_naive(W, X, y, reg):
"""
Structured SVM loss function, naive implementation (with loops).

Inputs have dimension D, there are C classes, and we operate on minibatches
of N examples.

Inputs:
- W: A numpy array of shape (D, C) containing weights.
- X: A numpy array of shape (N, D) containing a minibatch of data.
- y: A numpy array of shape (N,) containing training labels; y[i] = c means
that X[i] has label c, where 0 <= c < C.
- reg: (float) regularization strength

Returns a tuple of:
- loss as single float
- gradient with respect to weights W; an array of same shape as W
"""
dW = np.zeros(W.shape) # initialize the gradient as zero (3073,10)

# compute the loss and the gradient
num_classes = W.shape[1]    #10类
num_train = X.shape[0]      # 500个训练样本
loss = 0.0
for i in xrange(num_train):
scores = X[i].dot(W)        # (1,10)
correct_class_score = scores[y[i]] #第y[i]类的得分 或 第i个样本的所属类别的得分
for j in xrange(num_classes):
if j == y[i]:
continue            #跳出当前循环，执行下一次循环
margin = scores[j] - correct_class_score + 1 # note delta = 1
if margin > 0:
loss += margin          #计算data loss
dW[:,j] += X[i].T       # dW: 3073x10, 此处的值为3073x1,放在dW第j列，dW[:,j]只需加一次，外循环再循环500次
dW[:,y[i]] += -X[i].T   #dW[:,y[i]要加9次]，除非有margin<0;外循环再循环500次
# Right now the loss is a sum over all training examples, but we want it
# to be an average instead so we divide by num_train.
loss /= num_train         #由于样本太多，loss值累积，所以取均值
dW /= num_train
# Add regularization to the loss.
loss += reg * np.sum(W * W)       # L2 regularization
dW += reg * W
#############################################################################
# TODO:                                                                     #
# Compute the gradient of the loss function and store it dW.                #
# Rather that first computing the loss and then computing the derivative,   #
# it may be simpler to compute the derivative at the same time that the     #
# loss is being computed. As a result you may need to modify some of the    #
# code above to compute the gradient.                                       #
#############################################################################

return loss, dW

方法二 矢量化版：

def svm_loss_vectorized(W, X, y, reg):
"""
Structured SVM loss function, vectorized implementation.

Inputs and outputs are the same as svm_loss_naive.
"""
loss = 0.0
dW = np.zeros(W.shape) # initialize the gradient as zero

#############################################################################
# TODO:                                                                     #
# Implement a vectorized version of the structured SVM loss, storing the    #
# result in loss.                                                           #
#############################################################################
scores = X.dot(W)
correct = []
for i in xrange(X.shape[0]):
correct.append(scores[i,y[i]])
#或： correct = scores[range(500),list(y)]
correct = np.mat(correct).T
margin = np.maximum(0,scores - correct + 1)
loss = np.sum(margin) + np.sum(correct) - X.shape[0]  #将上面naive中j==y[i]多减多加的部分还原
#处理2：也可以将j==y[i]的项置0
#即：margin[range(500),list(y)]=0
loss = loss/X.shape[0]
loss += reg * np.sum(W * W)
#############################################################################
#                             END OF YOUR CODE                              #
#############################################################################

#############################################################################
# TODO:                                                                     #
# Implement a vectorized version of the gradient for the structured SVM     #
# loss, storing the result in dW.                                           #
#                                                                           #
# Hint: Instead of computing the gradient from scratch, it may be easier    #
# to reuse some of the intermediate values that you used to compute the     #
# loss.                                                                     #
#############################################################################
coeff = np.zeros(margin.shape) #500x10
coeff[margin>0]=1  #置1的目的是dWj这一项，让所有样本的Xi相加;而margin<0时不计入loss，所以置0
coeff[range(margin.shape[0]),list(y)] = 0 #将j==y[i]的项=0，因为求loss function时要求j!=y[i],所以这一项置0
coeff[range(margin.shape[0]),list(y)] = -np.sum(coeff,axis=1) #等号后面是(500,);dWyi要计算9次，除非有margin<0的情况
dW = (X.T).dot(coeff)
dW = dW/margin.shape[0] + reg*W
#############################################################################
#                             END OF YOUR CODE                              #
#############################################################################

return loss, dW

4. 检查梯度

用numerical gradient来检查上面求的analytic gradient

def grad_check_sparse(f, x, analytic_grad, num_checks=10, h=1e-5):
"""
sample a few random elements and only return numerical
in this dimensions.
"""
'''
@author: mckee    2018-1-8
f: lambda函数，返回loss
x: W (3073,10)
analytic_grad: grad
'''
for i in xrange(num_checks):
ix = tuple([randrange(m) for m in x.shape]) #x.shape = (3073,10)
print ('ix:',ix)
oldval = x[ix]
x[ix] = oldval + h # increment by h
fxph = f(x) # evaluate f(x + h)
x[ix] = oldval - h # increment by h
fxmh = f(x) # evaluate f(x - h)
x[ix] = oldval # reset

grad_numerical = (fxph - fxmh) / (2 * h)
grad_analytic = analytic_grad[ix]
rel_error = abs(grad_numerical - grad_analytic) / (abs(grad_numerical) + abs(grad_analytic))
print('numerical: %f analytic: %f, relative error: %e' % (grad_numerical, grad_analytic, rel_error))

5. Stochastic Gradient Descent

更新权值和loss

def train(self, X, y, learning_rate=1e-3, reg=1e-5, num_iters=100,
batch_size=200, verbose=False):
"""
Train this linear classifier using stochastic gradient descent.

Inputs:
- X: A numpy array of shape (N, D) containing training data; there are N
training samples each of dimension D.
- y: A numpy array of shape (N,) containing training labels; y[i] = c
means that X[i] has label 0 <= c < C for C classes.
- learning_rate: (float) learning rate for optimization.
- reg: (float) regularization strength.
- num_iters: (integer) number of steps to take when optimizing
- batch_size: (integer) number of training examples to use at each step.
- verbose: (boolean) If true, print progress during optimization.

Outputs:
A list containing the value of the loss function at each training iteration.
"""
num_train, dim = X.shape
num_classes = np.max(y) + 1 # assume y takes values 0...K-1 where K is number of classes
if self.W is None:
# lazily initialize W
print ('initialize W')
self.W = 0.001 * np.random.randn(dim, num_classes)

# Run stochastic gradient descent to optimize W
loss_history = []
for it in xrange(num_iters):
X_batch = None
y_batch = None

#########################################################################
# TODO:                                                                 #
# Sample batch_size elements from the training data and their           #
# corresponding labels to use in this round of gradient descent.        #
# Store the data in X_batch and their corresponding labels in           #
# y_batch; after sampling X_batch should have shape (dim, batch_size)   #
# and y_batch should have shape (batch_size,)                           #
#                                                                       #
# Hint: Use np.random.choice to generate indices. Sampling with         #
# replacement is faster than sampling without replacement.              #
#########################################################################
mask = np.random.choice(num_train,batch_size,replace = False)
X_batch = X[mask]
y_batch = y[mask]
#########################################################################
#                       END OF YOUR CODE                                #
#########################################################################

# evaluate loss and gradient
loss, grad = self.loss(X_batch, y_batch, reg)
loss_history.append(loss)

# perform parameter update
#########################################################################
# TODO:                                                                 #
# Update the weights using the gradient and the learning rate.          #
#########################################################################
self.W += -learning_rate * grad
#########################################################################
#                       END OF YOUR CODE                                #
#########################################################################
if it == 0:
print('initial loss:',loss)
if verbose and it % 100 == 0:
print('iteration %d / %d: loss %f' % (it, num_iters, loss))
if it == num_iters - 1:
print('iteration %d / %d: loss %f' % (it, num_iters, loss))

return loss_history

6. 评定精度

def predict(self, X):
"""
Use the trained weights of this linear classifier to predict labels for
data points.

Inputs:
- X: A numpy array of shape (N, D) containing training data; there are N
training samples each of dimension D.

Returns:
- y_pred: Predicted labels for the data in X. y_pred is a 1-dimensional
array of length N, and each element is an integer giving the predicted
class.
"""
y_pred = np.zeros(X.shape[0])
###########################################################################
# TODO:                                                                   #
# Implement this method. Store the predicted labels in y_pred.            #
###########################################################################
score = X.dot(self.W)
#print (score.shape)
#print (score[0].shape)
index = np.zeros(X.shape[0])
index = np.argmax(score,axis = 1)
y_pred = index
###########################################################################
#                           END OF YOUR CODE                              #
###########################################################################
return y_pred

得到预测label后，计算精度：

y_train_pred = svm.predict(X_train)
print('training accuracy: %f' % (np.mean(y_train == y_train_pred), ))#这样计算非常简洁、优雅
y_val_pred = svm.predict(X_val)
print('validation accuracy: %f' % (np.mean(y_val == y_val_pred), ))

三、涉及到的numpy重要函数

1.np.mean(matrix,axis)

功能：求均值，第二个参数可选

2.np.random.randn(a,b)

功能：从标准正态分布中返回一个或多个样本值，一个参数则为返回的样本个数，2个参数则为矩阵



3. np.maximum(a,b)

功能：比较a和b的大小

4.np.stack()、 np.vstack()、 np.hstack()

功能：数组、矩阵的堆叠、拼接