HDFS banlancer重平衡

一、优化参数

1.设置线程数

指定用于在DataNode间传输block数据的最大线程数

dfs.datanode.max.transfer.threads = 4096

（装有HBASE时可适当提升此值，老版本为dfs.datanode.max.xcievers）

2.设置平衡带宽

指定DataNode用于balancer的带宽，默认是1048576(1MB)单位为Byte，通常可设置为10MB；若机器性能优异可以设定为30-50MB；如果机器的网卡和交换机的带宽有限，可以设为1-10MB

dfs.datanode.balance.bandwidthPerSec = 31457280

3.设置balance线程数

指定DataNode上同时用于balance待移动block的最大线程个数，这个值默认是5

dfs.datanode.balance.max.concurrent.moves = 50

二、运行balancer

1.balancer理论步骤

1) 从namenode获取datanode磁盘的使用情况
2) 计算需要把哪些数据移动到哪些节点
3) 分别移动，完成后删除旧的block信息
4) 循环执行，直到达到平衡标准

2.执行balancer

#临时设置带宽（10MB）
sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 10240000

#各节点存储的浮动比例10%上下浮动
sudo -u hdfs hdfs balancer -threshold 10

3.监控日志

1）出现报错日志

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby.

此报错信息通常出现在配置了HA的HDFS中，balancer执行时不知道哪个namenode为active哪个为standby，当连接到standby的namenode时会出现以下报错。无须担心只要集群中为一个active状态的namenode，进程会尝试连接active状态的namenode，程序能够正常执行。

2）程序继续正常执行

INFO - Jul 28, 2017 3:32:04 PM          32             23.39 GB            22.80 GB              30 GB
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.27:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.22:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.23:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.25:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.34:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.29:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.39:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.32:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.33:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.28:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.26:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.37:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack1/192.168.111.24:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.35:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.38:50010
17/07/28 15:32:13 INFO net.NetworkTopology: Adding a new node: /rack2/192.168.111.36:50010
17/07/28 15:32:13 INFO balancer.Balancer: 1 over-utilized: [192.168.111.37:50010:DISK]
17/07/28 15:32:13 INFO balancer.Balancer: 3 underutilized: [192.168.111.25:50010:DISK, 192.168.111.35:50010:DISK, 192.168.111.36:50010:DISK]
17/07/28 15:32:13 INFO balancer.Balancer: Need to move 22.58 GB to make the cluster balanced.
17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.111.37:50010:DISK to 192.168.111.35:50010:DISK
17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 10 GB bytes from 192.168.111.27:50010:DISK to 192.168.111.25:50010:DISK
17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 7.83 GB bytes from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK
17/07/28 15:32:13 INFO balancer.Balancer: Decided to move 2.17 GB bytes from 192.168.111.32:50010:DISK to 192.168.111.36:50010:DISK
17/07/28 15:32:13 INFO balancer.Balancer: Will move 30 GB in this iteration
17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1079961644_6305194 with size=41 from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK through 192.168.111.39:50010
17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1079963221_6306771 with size=41 from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK through 192.168.111.39:50010
17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1075009112_1268291 with size=99980 from 192.168.111.27:50010:DISK to 192.168.111.25:50010:DISK through 192.168.111.27:50010
17/07/28 15:32:13 INFO balancer.Dispatcher: Successfully moved blk_1079960720_6304270 with size=5234467 from 192.168.111.39:50010:DISK to 192.168.111.36:50010:DISK through 192.168.111.39:50010