论文笔记——Deep Model Compression Distilling Knowledge from Noisy Teachers
2017-10-12 00:22
1141 查看
论文地址:https://arxiv.org/abs/1610.09650
注意:加噪声的时候对输入进行了采样,不然直接全加也太暴力了吧。
主要思想
这篇文章就是用teacher-student模型,用一个teacher模型来训练一个student模型,同时对teacher模型的输出结果加以噪声,然后来模拟多个teacher,这也是一种正则化的方法。1. teacher输出的结果加噪声以后,然后和student的输出结果计算L2 loss,作为student网络的反馈。
2. 加噪声
3. 计算L2 loss
4. 反向传播,更新参数
5. 算法过程
注意:加噪声的时候对输入进行了采样,不然直接全加也太暴力了吧。
等价于基于噪声的回归
实验结果
1. 对比了不同噪声比例对结果的影响,其实就是调参的过程。
2. 比较了学生加噪声和教师加噪声,结果是教师加噪声效果更加明显。
3. 比较了教师加噪声和一般的正则化操作(dropout)
总结
本文想法比较简单,就是给teacher输出结果加噪声,美曰其名,learn from multi teachers.相关文章推荐
- Face Model Compression by Distilling Knowledge from Neurons 论文理解
- 论文笔记《A Survey of Model Compression and Acceleration for Deep Neural Networks》
- 论文《From Facial Parts Responses to Face Detection: A Deep Learning Approach》笔记
- 读论文-Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model
- Distilling the Knowledge in a Neural Network[论文阅读笔记]
- [深度学习论文笔记][Depth Estimation] Depth Map Prediction from a Single Image using a Multi-Scale Deep Netw
- 论文笔记-深度估计(1)Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- 【论文笔记】Deep Learning Face Representation from Predicting 10,000 Classes
- 深度学习论文笔记-Deep Learning Face Representation from Predicting 10,000 Classes
- 深度学习论文笔记--Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- 人脸识别方向论文笔记(1)-- A Light CNN for Deep Face Representation With Noisy Labels
- 【论文笔记】From Facial Parts Responses to Face Detection: A Deep Learning Approach
- 论文笔记:Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- 论文笔记:Tag-Aware Personalized Recommendation Using a Hybrid Deep Model
- 【翻译+原创】Deep Learning Face Representation from Predicting 10,000 Classes 论文笔记
- 深度学习论文阅读笔记--Deep Learning Face Representation from Predicting 10,000 Classes
- 论文笔记——A Deep Neural Network Compression Pipeline: Pruning, Quantization, Huffman Encoding
- 【论文笔记】An End-to-End Model for QA over KBs with Cross-Attention Combining Global Knowledge
- 【论文笔记】Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- [深度学习论文笔记][Image Classification] ImageNet Classification with Deep Convolutional Neural Networks