论文阅读之Compressing Deep Convolutional Networks using Vector Quantization

1、论文索引

Y. Gong, L. Liu, M. Yang, and L. D. Bourdev, “Compressing deep convolutional networks using vector quantization,” CoRR, vol. abs/1412.6115, 2014.

2、论文摘要

  Deep convolutional neural networks (CNN) has become the most promising method for object recognition, repeatedly demonstrating record breaking results for image classification and object detection in recent years. However, a very deep CNN generally involves many layers with millions of parameters, making the storage of the network model to be extremely large. This prohibits the usage of deep CNNs on resource limited hardware, especially cell phones or other embedded devices. In this paper, we tackle this model storage issue by investigating information theoretical vector quantization methods for compressing the parameters of CNNs. In particular, we have found in terms of compressing the most storage demanding dense connected layers, vector quantization methods have a clear gain over existing matrix factorization methods. Simply applying k-means clustering to the weights or conducting product quantization can lead to a very good balance between model size and recognition accuracy. For the 1000-category classification task in the ImageNet challenge, we are able to achieve 16-24 times compression of the network with only 1% loss of classification accuracy using the state-of-the-art CNN.

  深度卷积神经网络(CNN)已成为最具前景的对象识别方法，近年来不断刷新图像分类和目标检测的纪录。然而，一个优秀的CNN通常涉及数百万个参数，这使得网络模型的存储空间变得非常大。这阻碍了在资源有限的硬件上使用深度CNNs，尤其是手机或其他嵌入式设备。本文研究了压缩CNNs参数的信息理论向量量化方法，从而解决了这个模型存储问题。特别地，我们发现在压缩最密集的全连接层时，矢量量化方法比现有的矩阵分解方法有明显的增益。简单地应用k-均值聚类对权重进行量化，可以在模型大小和识别精度之间取得很好的平衡。对于ImageNet挑战中1000个类别的分类任务，我们能够实现16-24倍的网络压缩，而使用最先进的CNN，只有1%的分类精度损失。

3、论文相关的工作

3.1、论文的贡献

1、第一个系统化的发掘和应用向量的量化方法对密集连接的全连接层进行压缩；

2、对不同的矩阵量化方法进行评估，表明结构化的量化尤其是product quantization方法比其他量化方法的优势

3、在其他的任务上作实验来证明压缩的模型具有广泛的应用性。

3.2、全连接层的压缩方法

1、矩阵分解方法SVD