pytorch给同一个layer的weight和bias设置不同的学习速率

经过在pytorch论坛的提问https://discuss.pytorch.org/t/how-to-set-different-learning-rate-for-weight-and-bias-in-one-layer/13450，现在总结如下：

1.使用dict简单粗暴设置，适用于层数较少的模型

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.layer = nn.Linear(1, 1)
self.layer.weight.data.fill_(1)
self.layer.bias.data.fill_(1)

def forward(self, x):
return self.layer(x)

if __name__=="__main__":
net = Net()
optimizer = optim.Adam([
{'params': net.layer.weight},
{'params': net.layer.bias, 'lr': 0.01}
], lr=0.1, weight_decay=0.0001)
out = net(Variable(torch.Tensor([[1]])))
out.backward()
optimizer.step()
print("weight", net.layer.weight.data.numpy(), "grad", net.layer.weight.grad.data.numpy())
print("bias", net.layer.bias.data.numpy(), "grad", net.layer.bias.grad.data.numpy())

若使用nn.Sequential可能会出现问题net.layer1.0.weight报语法错误，这时候使用net.layer1[0].weight代替即可。

2.使用代码生成dict列表，适用于层数多的layer

例子参考github上wkentaro/pytorch-fcn/blob/master/examples/voc/train_fcn32s.py#L105

其中函数get_parameter()来生成dict

def get_parameters(model, bias=False):
import torch.nn as nn
modules_skipped = (
nn.ReLU,
nn.MaxPool2d,
nn.Dropout2d,
nn.Sequential,
torchfcn.models.FCN32s,
torchfcn.models.FCN16s,
torchfcn.models.FCN8s,
)
for m in model.modules():
if isinstance(m, nn.Conv2d):
if bias:
yield m.bias
else:
yield m.weight
elif isinstance(m, nn.ConvTranspose2d):
# weight is frozen because it is just a bilinear upsampling
if bias:
assert m.bias is None
elif isinstance(m, modules_skipped):
continue
else:
raise ValueError('Unexpected module: %s' % str(m))

应该注意根据情况调整modules_skipped的内容，应用方法如下：

optimizer = optim.SGD([
{'params': get_parameters(model, bias=False)},
{'params': get_parameters(model, bias=True),
'lr': opt.lr * 0.1}
], lr=opt.lr, weight_decay=opt.weight_decay)

但这两个方法都有一个问题，当我的学习速率要随着训练改变的时候，比如step，还不知道要怎么重新加载学习速率，现在是简单粗暴手动修改+load前一次的训练结果来进行的，如何实现类似caffe自动step学习速率的功能，目前没有头绪，如果谁有头绪，求教。