Local Response Normalization (LRN)
2017-03-27 14:20
531 查看
This concept was raised in AlexNet, click here to learn more.
Local response normalization algorithm was inspired by the real neurons, as the author said, “bears some resemblance to the local contrast normalization”. The common point is that they both want to introduce competitions to the neuron outputs, the difference is LRN do not subtract mean and the competition happens among the outputs of adjacent kernels at the same layer.
The formula for LRN is as follows:
a(i, x, y) represents the i th conv. kernel’s output (after ReLU) at the position of (x, y) in the feature map.
b(i, x, y) represents the output of local response normalization, and of course it’s also the input for the next layer.
N is the number of the conv. kernel number.
n is the adjacent conv. kernel number, this number is up to you. In the article they choose n = 5.
k, α, β are hyper-parameters, in the article, they choose k = 2, α = 10e-4, β = 0.75.
Flowchart of Local Response Normalization
I drew the above figure to illustrate the process of LRN in neural network. Just a few tips here:
After knowing what LRN is, another question is: what the output of LRN looks like?
Because the LRN happens after ReLU, so the inputs should all be no less than 0. The following graph tries to give you an intuitive understanding on the output of LRN, however, you still need to use your imagination.
Be noted that the x axis represents the summation of the squared output of ReLU, ranging from 0 to 1000, and the y axis represents b(i, x, y) divides a(i, x, y). The hyper-parameters are set default to the article.
So, the real b(i, x, y)’s value should be the the y axis’s value multiplied with the a(i, x, y), use your imagination here, two different inputs a(i, x, y) pass through this function. Since the slope at the beginning is very steep, little difference among the inputs will be significantly enlarged, this is where the competition happens.
The figure was generated by the following python code:
1
2
3
4
5
6
7
8
9
10
11
import numpy as np
import matplotlib.pyplot as plt
def lrn(x):
… y = 1 / (2 + (10e-4) * x * 2) * 0.75
… return y
input = np.arange(0, 1000, 0.2)
output = lrn(input)
plt.plot(input, output)
plt.xlabel(‘sum(x^2)’)
plt.ylabel(‘1 / (k + a * sum(x^2))’)
plt.show()
Local response normalization algorithm was inspired by the real neurons, as the author said, “bears some resemblance to the local contrast normalization”. The common point is that they both want to introduce competitions to the neuron outputs, the difference is LRN do not subtract mean and the competition happens among the outputs of adjacent kernels at the same layer.
The formula for LRN is as follows:
a(i, x, y) represents the i th conv. kernel’s output (after ReLU) at the position of (x, y) in the feature map.
b(i, x, y) represents the output of local response normalization, and of course it’s also the input for the next layer.
N is the number of the conv. kernel number.
n is the adjacent conv. kernel number, this number is up to you. In the article they choose n = 5.
k, α, β are hyper-parameters, in the article, they choose k = 2, α = 10e-4, β = 0.75.
Flowchart of Local Response Normalization
I drew the above figure to illustrate the process of LRN in neural network. Just a few tips here:
* This graph presumes that the i th kernel is not at the edge of the kernel space. If i equals zero or one or last or one to the last, one or two additional zero padding conv. kernels are required. * In the article, n is 5, we presume n/2 is integer division, 5/2 = 2. * Summation of the squares of output of ReLU stands for: for each output of ReLU, compute its square, then, add the 5 squared value together. This process is the summation term of the formula. * I presume the necessary padding is used by the input feature map so that the output feature maps have the same size of the input feature map, if you really care. But this padding may not be quite necessary.
After knowing what LRN is, another question is: what the output of LRN looks like?
Because the LRN happens after ReLU, so the inputs should all be no less than 0. The following graph tries to give you an intuitive understanding on the output of LRN, however, you still need to use your imagination.
Be noted that the x axis represents the summation of the squared output of ReLU, ranging from 0 to 1000, and the y axis represents b(i, x, y) divides a(i, x, y). The hyper-parameters are set default to the article.
So, the real b(i, x, y)’s value should be the the y axis’s value multiplied with the a(i, x, y), use your imagination here, two different inputs a(i, x, y) pass through this function. Since the slope at the beginning is very steep, little difference among the inputs will be significantly enlarged, this is where the competition happens.
The figure was generated by the following python code:
1
2
3
4
5
6
7
8
9
10
11
import numpy as np
import matplotlib.pyplot as plt
def lrn(x):
… y = 1 / (2 + (10e-4) * x * 2) * 0.75
… return y
input = np.arange(0, 1000, 0.2)
output = lrn(input)
plt.plot(input, output)
plt.xlabel(‘sum(x^2)’)
plt.ylabel(‘1 / (k + a * sum(x^2))’)
plt.show()
相关文章推荐
- LRN ( Local Response Normalization) 局部响应归一化层
- Local Response Normalization作用——感觉LRN也是解决的梯度消失和爆炸问题,统计机器学习中的一个经典假设是“源空间(source domain)和目标空间(target domain)的数据分布(distribution)是一致的
- caffe中的Local Response Normalization (LRN)有什么用,和激活函数区别
- 深度学习的局部响应归一化LRN(Local Response Normalization)理解
- 深度学习: 局部响应归一化 (Local Response Normalization,LRN)
- LRN (Local Response Normalization,即局部响应归一化层)
- ImageNet中的LRN(Local Response Normalization)
- local_response_normalization 和 batch_normalization
- 局部响应归一化(Local Response Normalization)
- 【TensorFlow】tf.nn.local_response_normalization详解,lrn正则法如何计算?
- 基于FPGA的GoogleNet加速器-Local Response Normalization
- local response normalization/batch normalization
- Save soapui response to local
- Human Pose Estimation using Global and Local Normalization
- svhn的local contrast normalization处理
- 软件随想录(local.joelonsoftware.com/wiki)-2000年07月25日 匿名回应 - Anonymous Response
- 姿态论文整理--04-Human pose estimation using global and local normalization
- Fiddler AutoResponse redir 高级使用
- Passing address of non-local object to _autoreleasing parameter for write-back
- Correct use of System.Web.HttpResponse.Redirect