Mellanox ConnectX-5 Host Chaining


Turn off Single Root I/O Virtualization (SR-IOV)

Boot into the host BIOS and disable SRIOV. Typically found in the PCI settings.

Single Root IO Virtualization (SR-IOV) is a technology that allows a physical PCIe device to present itself multiple times through the PCIe bus. This technology enables multiple virtual instances of the device with separate resources. NVIDIA adapters are capable of exposing up to 127 virtual instances (Virtual Functions (VFs) for each port in the NVIDIA ConnectX® family cards. These virtual functions can then be provisioned separately. Each VF can be seen as an additional device connected to the Physical Function. It shares the same resources with the Physical Function, and its number of ports equals those of the Physical Function. SR-IOV is commonly used in conjunction with an SR-IOV enabled hypervisor to provide virtual machines direct hardware access to network resources hence increasing its performance.


Hardware Configuration

Currently we've only been successful with a ring topology.


When host chaining is set up, all nodes in the chain need to be powered up in order for connections to pass through to other nodes.
If you disable interfaces in the OS, packets can still pass through the NIC to other nodes.
If the host OS is not running, packets will NOT be forwarded to other nodes, effectively breaking connections to downsteam hosts

Install MLNX EN drivers

For host chaining, the EN driver is recommended over the OFED package.

Mellanox Downloads

Mellanox Linux Drivers:

5.9- Release:

Release Notes:

Driver (Ubuntu 22.04):

User Manual:

Install EN driver

# Download driver

# Untar and change into driver directory
tar -xzf MLNX_EN-5.9-
cd MLNX_EN-5.9-

# To view options
./install --help

# Install the Ethernet drivers only
./install --without-fw-update --eth-only

# If your adapter requires a firmware update you can leave out the --without-fw-update flag
./install --eth-only

# Restart the mlnx-en.d service once the install is complete
/etc/init.d/mlnx-en.d restart

Get device path

root@host:~# lspci | grep Mellanox
4b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
4b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

Note the device path, it will be need for future commands.

In the example our device path is `4b:00.0`

Alternate option

You can also use `mstconfig q` to find the path

root@host:/tmp# mstconfig q | head

Device #1:

Device type:    ConnectX5
Name:           MCX512A-ACA_Ax_Bx
Description:    ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; tall bracket; ROHS R6
Device:         /sys/bus/pci/devices/0000:4b:00.0/config

Configurations:                              Next Boot

Verify Firmware Version

Use the `mstflint` command to verify FW versions.

root@host:/tmp# mstflint -d 4b:00.0 q
Image type:            FS4
FW Version:            16.35.2000
FW Release Date:       24.11.2022
Product Version:       16.35.2000
Rom Info:              type=UEFI version=14.28.16 cpu=AMD64
                       type=PXE version=3.6.805 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             e8ebd303001b9a28        4
Base MAC:              e8ebd31b9a28            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000000080
Security Attributes:   N/A

Set host chaining mode on the NIC:

Here we set `HOST_CHAINING_MODE` to enabled, we also disable `SRIOV_EN` functionality

root@host:~# mstconfig -d 4b:00.0 set HOST_CHAINING_MODE=1 SRIOV_EN=0

Device #1:

Device type:    ConnectX5       
Name:           MCX512A-ACA_Ax_Bx
Description:    ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; tall bracket; ROHS R6
Device:         4b:00.0         

Configurations:                              Next Boot       New

         HOST_CHAINING_MODE                  DISABLED(0)     BASIC(1)        
         SRIOV_EN                            True(1)         False(0)

Apply new Configuration? (y/n) [n] : yes
Applying... Done!
-I- Please reboot machine to load new configurations.

Restart NIC


Reboot the server for the changes to take effect


Use the `mstfwreset` command to restart the firmware on the controller.

root@host:~# mstfwreset -d 4b:00.0 reset

Set IP address on Mellanox interfaces

Host chaining seems to work whether you add an IP to only one interface or both. It is recommended if you use both interfaces that you use unique IP addresses for each.

Be sure to set MTU to 9000 to enable jumbo frames and maximize bandwidth.

# Example netplan configuration
  version: 2
  renderer: networkd
        macaddress: 12:34:56:78:90:AB
      dhcp4: no
      dhcp6: no
      mtu: 9000
      addresses: [ ]
      link-local: []
        macaddress: 12:34:56:78:90:AC
      dhcp4: no
      dhcp6: no
      mtu: 9000
      addresses: [ ]
      link-local: []

Verify you connectivity

`iperf` is a handy utility to verify bandwidth speed across links.

Host node

`--bind` is option, but allows you to force an interface to use.
root@host:~# iperf -s --bind
Server listening on TCP port 5001
TCP window size:  128 KByte (default)

Client node

root@client:~# iperf -c
Client connecting to, TCP port 5001
TCP window size:  325 KByte (default)
[  1] local port 39962 connected with port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0077 sec  28.6 GBytes  24.5 Gbits/sec
Here you can see the connection was measured at 24.5 Gb/s

Misc Commands

Check driver info with modinfo

root@host:~# modinfo mlx5_core
filename:       /lib/modules/5.15.0-67-generic/updates/dkms/mlx5_core.ko
alias:          auxiliary:mlx5_core.eth-rep
alias:          auxiliary:mlx5_core.eth
basedon:        Korg 6.0-rc4
version:        5.9-0.5.5
license:        Dual BSD/GPL
description:    Mellanox 5th generation network adapters (ConnectX series) core driver
author:         Eli Cohen <>


Forum discussion on host chaining:

-- Tom Cauduro - 2023-04-12

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng RingTopo.png r1 manage 46.8 K 2023-04-12 - 13:56 TomCauduro  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2023-04-12 - TomCauduro
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback