DenseNet--Densely Connected Convolutional Networks

Compelling advantages

alleviate the vanishing-gradient problem

strengthen feature propagation

encourage feature reuse

substantially reduce the number of parameters.

requires fewer parameters

no need to relearn redundant feature-maps

observe that dense connections have a regularizing effect, which reduces overfitting on tasks with smaller training set sizes

easy to train

*

Contributions

maximum information flow between layers in the network, we connect all layers (with matching feature-map sizes) directly with each other.

we never combine features through summation before they are passed into a layer; instead, we combine features by concatenating them.

Architecture

Methods

identity function (may impede the information flow in the network)



Dense connectivity



Composite function

at lth composite layer: BN –> ReLU–>3 x 3 Conv

Pooling layers

1 x 1 conv –> 2 x 2 average pooling

Growth rate

lth layer has k0 + k ×(l −1)

Bottleneck layers

1×1 convolution can be introduced as bottleneck layer before each 3×3 convolution to reduce the number of input feature-maps, and thus to improve computational efficiency

Experiments

Others

ResNet: each layer reads the state from its preceding layer and writes to the subsequent layer. It changes the state but also passes on information that needs to be preserved. ResNets [11] make this information preservation explicit through additive identity transformations.

stochastic depth was proposed as a way to successfully train a 1202-layer ResNet [13]. Stochastic depth improves the training of deep residual networks by dropping layers randomly during training.