CS231n作业笔记2.5:dropout的实现与应用
2017-01-04 13:38
417 查看
CS231n简介
详见 CS231n课程笔记1:Introduction。本文都是作者自己的思考,正确性未经过验证,欢迎指教。
作业笔记
dropout中唯一需要注意的就是为了平衡train与test,通过除以期望值即可。1. 前向传播
if mode == 'train': mask = (np.random.rand(*x.shape)<p) out = x*mask / p elif mode == 'test': out = x mask = np.ones_like(x)
2. 后向传播
if mode == 'train': dx = dout * mask / dropout_param['p'] elif mode == 'test': dx = dout return dx
3. 应用:带dropout的多层神经网络
在每一层ReLU后接一层dropout即可。关于多层神经网络的实现,请参考CS231n作业笔记2.4:Batchnorm的实现与使用。cache = {} hidden_value = None hidden_value,cache['fc1'] = affine_forward(X,self.params['W1'],self.params['b1']) if self.use_batchnorm: hidden_value,cache['bn1'] = batchnorm_forward(hidden_value, self.params['gamma1'], self.params['beta1'], self.bn_params[0]) hidden_value,cache['relu1'] = relu_forward(hidden_value) if self.use_dropout: hidden_value, cache['drop1'] = dropout_forward(hidden_value,self.dropout_param) for index in range(2,self.num_layers): hidden_value,cache['fc'+str(index)] = affine_forward(hidden_value,self.params['W'+str(index)],self.params['b'+str(index)]) if self.use_batchnorm: hidden_value,cache['bn'+str(index)] = batchnorm_forward(hidden_value, self.params['gamma'+str(index)], self.params['beta'+str(index)], self.bn_params[index-1]) hidden_value,cache['relu'+str(index)] = relu_forward(hidden_value) if self.use_dropout: hidden_value, cache['drop'+str(index)] = dropout_forward(hidden_value,self.dropout_param) scores,cache['score'] = affine_forward(hidden_value,self.params['W'+str(self.num_layers)],self.params['b'+str(self.num_layers)]) # If test mode return early if mode == 'test': return scores loss, grads = 0.0, {} loss,dscores = softmax_loss(scores,y) for index in range(1,self.num_layers+1): loss += 0.5*self.reg*np.sum(self.params['W'+str(index)]**2) dhidden_value,grads['W'+str(self.num_layers)],grads['b'+str(self.num_layers)] = affine_backward(dscores,cache['score']) for index in range(self.num_layers-1,1,-1): if (self.use_dropout): dhidden_value = dropout_backward(dhidden_value, cache['drop'+str(index)]) dhidden_value = relu_backward(dhidden_value,cache['relu'+str(index)]) if self.use_batchnorm: dhidden_value, grads['gamma'+str(index)], grads['beta'+str(index)] = batchnorm_backward(dhidden_value, cache['bn'+str(index)]) dhidden_value,grads['W'+str(index)],grads['b'+str(index)] = affine_backward(dhidden_value,cache['fc'+str(index)]) if (self.use_dropout): dhidden_value = dropout_backward(dhidden_value, cache['drop1']) dhidden_value = relu_backward(dhidden_value,cache['relu1']) if self.use_batchnorm: dhidden_value, grads['gamma1'], grads['beta1'] = batchnorm_backward(dhidden_value, cache['bn1']) dhidden_value,grads['W1'],grads['b1'] = affine_backward(dhidden_value,cache['fc1']) for index in range(1,self.num_layers+1): grads['W'+str(index)] += self.reg * self.params['W'+str(index)]
相关文章推荐
- CS231n作业笔记2.1:两层全连接神经网络的分层实现
- CS231n课程作业(二) Multi-Layer Net、BN、Dropout、Optimization
- CS231n作业笔记2.2:多层神经网络的实现
- CS231n作业笔记2.6:卷积层以及池化层的实现
- MXNet动手学深度学习笔记:Gluon实现Dropout
- CS231n 2016 通关 第五、六章 Dropout 作业
- 『cs231n』作业2选讲_通过代码理解Dropout
- CS231n课程学习笔记(七)——数据预处理、批量归一化和Dropout
- CS231n作业笔记2.4:Batchnorm的实现与使用
- dropout 的快熟实现笔记 --tensorlayer
- CS231n笔记5--Weights Update 与 Dropout
- SMP3.0学习笔记之八 使用Kapsel Update插件实现移动应用的自动更新
- Android应用开发笔记(12):Android应用的自动升级、更新模块的实现
- iPhone开发笔记(19)实现类似网易新闻当应用状态变为active时手动调用下拉刷新的功能
- Android应用开发笔记(12):Android应用的自动升级、更新模块的实现
- 【电信增值业务学习笔记】9基于智能网的增值业务实现技术和应用
- JAVA学习笔记38——模拟实现Iterator+HashMap的“分拣”原理+“分拣”的应用
- Android应用开发笔记(12):Android应用的自动升级、更新模块的实现
- android应用开发揭秘examples_04-13笔记(Menu的2种实现方式)
- 笔记91--WindowManager.LayoutParams应用之实现悬浮窗口