搭建 Mellanox ConnectX-3 EN 10/40 Gbps 网卡 DPDK 运行环境

DPD v2.0 当前依赖Mlx版本

Mellanox OFED 2.4-1.

Firmware version 2.33.5000 and higher.

1> 编译 mlx4 libraries

http://dpdk.org/download/mlx4/

因要使用 ibv_xxx 系列工具， ibv_xxx 不建议再使用DPDK.org下载的 mlx4 libraries. 可使用 MLNX OFED 内置 mlx4 libraries. 必须采用动态编译方式，方能运行正确.
编译完 MLNX OFED mxl4 libraries , 使用 make install DESTDIR=xxx , 并打包.

2> 编译DPDK v2.0.0

3> 编译mlx4 ofed 内核模块及mstflint程序

http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

注意 MLNX_OFED 2.4-1.0.0 只支持以下OS与内核版本

- RHEL/CentOS 6.3 x86_64

- RHEL/CentOS 6.4 x86_64/PPC

- RHEL/CentOS 6.5 x86_64/PPC

- RHEL/CentOS 6.6 x86_64/PPC

- RHEL/CentOS 7.0 x86_64/PPC

- SLES11 SP2 x86_64/PPC

- SLES11 SP3 x86_64/PPC

- SLES12 x86_64

- OEL 6.4 x86_64

- OEL 6.5 x86_64

- OEL 6.6 x86_64

- Citrix XenServer Host 6.2 i686

- Fedora 19 x86_64

- Fedora 21 x86_64

- Ubuntu 12.04.4 x86_64

- Ubuntu 14.04 x86_64/PPC4le

- Ubuntu 14.10 x86_64/PPC4le

- Debian 6.0.9 x86_64

- Debian 7.5 x86_64

- Debian 7.6 x86_64

- kernel 3.10.48

- kernel 3.11.10

- kernel 3.14.12

- kernel 3.15.5

- kernel 3.16

- kernel 3.17

- kernel 3.18

Note: 建议使用RHEL相关的内核, 直接可以使用RPM安装相应模块.

编译mlx4内核模块

tar -zxvf mlnx-ofa_kernel-2.4.tgz

cd mlnx-ofa_kernel-2.4

# 配置，当前采用的是RHEL7第一版内核, 可用 yumdownloader --source kernel-3.10.0-123 下载.

./configure --with-mlx4_fc-mod --with-mlx4_inf-mod --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mlx4-mod --with-mlx4_en-mod --with-ipoib-mod --with-srp-mod --with-rds-mod --with-iser-mod --kernel-version=3.10.0-mlx --modules-dir=/lib/modules/3.10.0-mlx/ --kernel-sources=/work/kernel-3.10.0-123.el7.src/linux-3.10.0-123.el7

make

make install # 建议安装，方便建立内核依赖关系

Note: 如为自定义内核配置，CONFIG_INFINIBAND 确认为 'n' 或者 'm', 否则此模块打包进内核在加载mlx4模块时会提示符号冲突.

编译 mft 内核模块，mstflint 使用.

tar -zxvf kernel-mft-3.8.0.tgz

cd kernel-mft-3.8.0

make KPVER=/work/kernel-3.10.0-123.el7.src/linux-3.10.0-123.el7

得到 mst_pciconf.ko mst_pci.ko

编译 mstflint

tar -zxvf mstflint-3.8.0-1.27.gf3d39b6.tar.gz

cd mstflint-3.8.0

./configure --disable-inband --prefix=/usr

make install

4 . 加载模块
将模块打包并安装到目标机器下

modprobe ib_uverbs
modprobe mlx4_en
modprobe mlx4_core
modprobe mlx4_ib log_num_mgm_entry_size=-1
# 加载mst相关内核
insmod mst_pci.ko
insmod mst_pciconf.ko

运行 ibv_devinfo 获取设备信息

# ibv_devinfo -v
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.33.5100
node_guid: 0002:c903:00fa:a480
sys_image_guid: 0002:c903:00fa:a483
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090110019
phys_port_cnt: 2
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffe00
max_qp: 392632
max_qp_wr: 16351
device_cap_flags: 0x005c9c76
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
UD_***_PORT_ENFORCE
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
XRC
Unknown flags: 0x004c8000
device_cap_exp_flags: 0xD122001600000000
EXP_DEVICE_QPG
EXP_UD_RSS
EXP_MEM_WINDOW
EXP_MEM_MGT_EXTENSIONS
EXP_MW_TYPE_2B
EXP_CROSS_CHANNEL
EXP_MR_ALLOCATE
EXP_SHARED_MR
EXT_ATOMICS
Unknown flags: 0x8000000000000000
max_sge: 32
max_sge_rd: 0
max_cq: 65408
max_cqe: 4194303
max_mr: 524032
max_pd: 32764
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 6282112
max_qp_init_rd_atom: 128
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
log atomic arg sizes (mask) 8
max fetch and add bit boundary 64
log max atomic inline 3
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 131072
max_mcast_qp_attach: 244
max_total_mcast_qp_attach: 31981568
max_ah: 2147483647
max_fmr: 0
max_srq: 65472
max_srq_wr: 16383
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 15
hca_core_clock: 427
max_klm_list_size: 0
max_send_wqe_inline_klms: 0
max_umr_recursion_depth: 0
max_umr_stride_dimension: 0
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
dc_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
raw_eth_odp_caps:
NO SUPPORT
max_dct: 0
port: 1
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x40000000
port_cap_flags: 0x00010000
max_vl_num: 2 (2)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 1
gid_tbl_len: 128
subnet_timeout: 0
init_type_reply: 0
active_width: 4X (2)
active_speed: 10.0 Gbps (4)
phys_state: DISABLED (3)
GID[ 0]: fe80:0000:0000:0000:0202:c9ff:fefa:a481

port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: InfiniBand
max_msg_sz: 0x40000000
port_cap_flags: 0x02504868
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 128
gid_tbl_len: 128
subnet_timeout: 0
init_type_reply: 0
active_width: 4X (2)
active_speed: 2.5 Gbps (1)
phys_state: POLLING (2)
GID[ 0]: fe80:0000:0000:0000:0002:c903:00fa:a482

ibv_devices 可获取设备相关 GUID

# ibv_devices
device node GUID
------ ----------------
mlx4_0 0002c90300faa480

connectx_port_config 在 mlnx-ofa_kernel-2.4-OFED.2.4.1.0.0.1 包的 ofed_scripts目录下，用于设置端口模式;

将端口配置成Ethernet模式;

# ./connectx_port_config

ConnectX PCI devices :
|----------------------------|
| 1 0000:02:00.0 |
|----------------------------|

Before port change:
auto (eth)
auto (ib)

|----------------------------|
| Possible port modes: |
| 1: Infiniband |
| 2: Ethernet |
| 3: AutoSense |
|----------------------------|
Select mode for port 1 (1,2,3): 2
Select mode for port 2 (1,2,3): 2

After port change:
eth
eth

ifconfig 查看新增网卡设备.

# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:1E:67:9C:D4:90
inet addr:192.168.1.37 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:872 errors:0 dropped:0 overruns:0 frame:0
TX packets:734 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:67141 (65.5 KiB) TX bytes:82711 (80.7 KiB)
Memory:d0b20000-d0b40000

eth1 Link encap:Ethernet HWaddr 00:1E:67:9C:D4:91
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Memory:d0b00000-d0b20000

eth2 Link encap:Ethernet HWaddr 00:02:C9:FA:A4:81
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth3 Link encap:Ethernet HWaddr 00:02:C9:FA:A4:82
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:225 errors:0 dropped:0 overruns:0 frame:0
TX packets:225 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:25200 (24.6 KiB) TX bytes:25200 (24.6 KiB)

查看新增网卡驱动属性.

ethtool -i eth2
driver: mlx4_en
version: 2.4-1.0.0 (Apr 22 2015)
firmware-version: 2.33.5100
bus-info: 0000:02:00.0
ethtool -i eth3
driver: mlx4_en
version: 2.4-1.0.0 (Apr 22 2015)
firmware-version: 2.33.5100
bus-info: 0000:02:00.0

由此mlx4加载成功.

5> 运行测试程序
./testpmd -c 0xffff -m 4096 -n 1 -r 1 -- --rx=1 --tx=2 -i

set fwd txonly
start all
show port stats all

./trafgen --port0 port0.conf

参考:
http://dpdk.org/doc/guides/nics/mlx4.html