Single Shot MultiBox Detector（MXNet）源码阅读笔记（2）

Single Shot MultiBox Detector（MXNet）源码阅读笔记（2）

初始化的方法Xavier

We initialize the parameters for all the newly added convolutional layers with the “xavier” method.

论文中所述，我们对新增加的卷积层使用“xavier”的方法进行初始化。

Xavier Initialization很简单，其实就是Var(Wi)=2nin+nout

nin-输入神经元的数量

nout-输出神经元的数量

则

scale=magnitudeVar(Wi)−−−−−−−−−−√

具体解释见andy’sblog

原理出处见论文Understanding the difficulty of training deep feedforward neural networks

MXnet中代码initializer.py部分

class Xavier(Initializer):
"""Initialize the weight with Xavier or similar initialization scheme.
Parameters
----------
rnd_type: str, optional
Use ```gaussian``` or ```uniform``` to init
factor_type: str, optional
Use ```avg```, ```in```, or ```out``` to init
magnitude: float, optional
scale of random number range
"""
def __init__(self, rnd_type="uniform", factor_type="avg", magnitude=3):
super(Xavier, self).__init__(rnd_type=rnd_type, factor_type=factor_type,magnitude=magnitude)
self.rnd_type = rnd_type
self.factor_type = factor_type
self.magnitude = float(magnitude)

def _init_weight(self, _, arr):
shape = arr.shape
hw_scale = 1.
if len(shape) > 2:
hw_scale = np.prod(shape[2:])
fan_in, fan_out = shape[1] * hw_scale, shape[0] * hw_scale
factor = 1.
if self.factor_type == "avg":
factor = (fan_in + fan_out) / 2.0
# 计算方差
elif self.factor_type == "in":
factor = fan_in
elif self.factor_type == "out":
factor = fan_out
else:
raise ValueError("Incorrect factor type")
scale = np.sqrt(self.magnitude / factor)
# 计算范围
if self.rnd_type == "uniform":
random.uniform(-scale, scale, out=arr)
elif self.rnd_type == "gaussian":
random.normal(0, scale, out=arr)
else:
raise ValueError("Unknown random type")