CF Web>Hardware>HostChaining (2023-04-12, TomCauduro)

Mellanox ConnectX-5 Host Chaining

Pre-requisites

Turn off Single Root I/O Virtualization (SR-IOV)

Boot into the host BIOS and disable SRIOV. Typically found in the PCI settings.

About SRIOV
Single Root IO Virtualization (SR-IOV) is a technology that allows a physical PCIe device to present itself multiple times through the PCIe bus. This technology enables multiple virtual instances of the device with separate resources. NVIDIA adapters are capable of exposing up to 127 virtual instances (Virtual Functions (VFs) for each port in the NVIDIA ConnectX® family cards. These virtual functions can then be provisioned separately. Each VF can be seen as an additional device connected to the Physical Function. It shares the same resources with the Physical Function, and its number of ports equals those of the Physical Function. SR-IOV is commonly used in conjunction with an SR-IOV enabled hypervisor to provide virtual machines direct hardware access to network resources hence increasing its performance.

Source: https://docs.nvidia.com/networking/pages/viewpage.action?pageId=111589463

Hardware Configuration

Currently we've only been successful with a ring topology.

Important
When host chaining is set up, all nodes in the chain need to be powered up in order for connections to pass through to other nodes.
If you disable interfaces in the OS, packets can still pass through the NIC to other nodes.
If the host OS is not running, packets will NOT be forwarded to other nodes, effectively breaking connections to downsteam hosts

Install MLNX EN drivers

For host chaining, the EN driver is recommended over the OFED package.

Mellanox Downloads

Mellanox Linux Drivers:

https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/

5.9-0.5.6.0 Release:

Release Notes:

https://docs.nvidia.com/networking/display/MLNXENv590560/Release+Notes

Driver (Ubuntu 22.04):

https://www.mellanox.com/downloads/ofed/MLNX_EN-5.9-0.5.6.0/mlnx-en-5.9-0.5.6.0-ubuntu22.04-x86_64.tgz

User Manual:

https://docs.nvidia.com/networking/display/MLNXENv590560/User+Manual

Install EN driver

# Download driver
wget https://www.mellanox.com/downloads/ofed/MLNX_EN-5.9-0.5.6.0/mlnx-en-5.9-0.5.6.0-ubuntu22.04-x86_64.tgz

# Untar and change into driver directory
tar -xzf MLNX_EN-5.9-0.5.6.0/mlnx-en-5.9-0.5.6.0-ubuntu22.04-x86_64.tgz
cd MLNX_EN-5.9-0.5.6.0/mlnx-en-5.9-0.5.6.0-ubuntu22.04-x86_64

# To view options
./install --help

# Install the Ethernet drivers only
./install --without-fw-update --eth-only

# If your adapter requires a firmware update you can leave out the --without-fw-update flag
./install --eth-only


# Restart the mlnx-en.d service once the install is complete
/etc/init.d/mlnx-en.d restart

Get device path

root@host:~# lspci | grep Mellanox
4b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
4b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

Note the device path, it will be need for future commands.

In the example our device path is `4b:00.0`

Alternate option

You can also use `mstconfig q` to find the path

root@host:/tmp# mstconfig q | head

Device #1:
----------

Device type:    ConnectX5
Name:           MCX512A-ACA_Ax_Bx
Description:    ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; tall bracket; ROHS R6
Device:         /sys/bus/pci/devices/0000:4b:00.0/config

Configurations:                              Next Boot

Verify Firmware Version

Use the `mstflint` command to verify FW versions.

root@host:/tmp# mstflint -d 4b:00.0 q
Image type:            FS4
FW Version:            16.35.2000
FW Release Date:       24.11.2022
Product Version:       16.35.2000
Rom Info:              type=UEFI version=14.28.16 cpu=AMD64
                       type=PXE version=3.6.805 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             e8ebd303001b9a28        4
Base MAC:              e8ebd31b9a28            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_0000000080
Security Attributes:   N/A

Set host chaining mode on the NIC:

Here we set `HOST_CHAINING_MODE` to enabled, we also disable `SRIOV_EN` functionality

root@host:~# mstconfig -d 4b:00.0 set HOST_CHAINING_MODE=1 SRIOV_EN=0

Device #1:

Device type:    ConnectX5       
Name:           MCX512A-ACA_Ax_Bx
Description:    ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; tall bracket; ROHS R6
Device:         4b:00.0         

Configurations:                              Next Boot       New

         HOST_CHAINING_MODE                  DISABLED(0)     BASIC(1)        
         SRIOV_EN                            True(1)         False(0)

Apply new Configuration? (y/n) [n] : yes
Applying... Done!
-I- Please reboot machine to load new configurations.

Restart NIC

Alternative

Use the `mstfwreset` command to restart the firmware on the controller.

root@host:~# mstfwreset -d 4b:00.0 reset

Set IP address on Mellanox interfaces

Host chaining seems to work whether you add an IP to only one interface or both. It is recommended if you use both interfaces that you use unique IP addresses for each.

Set MTU
Be sure to set MTU to 9000 to enable jumbo frames and maximize bandwidth.

# Example netplan configuration
network:
  version: 2
  renderer: networkd
  ethernets:
    ens2f0np0:
      match:
        macaddress: 12:34:56:78:90:AB
      dhcp4: no
      dhcp6: no
      mtu: 9000
      addresses: [ 192.168.1.1/24 ]
      link-local: []
    ens2f1np1:
      match:
        macaddress: 12:34:56:78:90:AC
      dhcp4: no
      dhcp6: no
      mtu: 9000
      addresses: [ 192.168.1.2/24 ]
      link-local: []

Verify you connectivity

`iperf` is a handy utility to verify bandwidth speed across links.

Host node

`--bind` is option, but allows you to force an interface to use.

root@host:~# iperf -s --bind 192.168.1.1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------

Client node

root@client:~# iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.101 port 39962 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0077 sec  28.6 GBytes  24.5 Gbits/sec

Here you can see the connection was measured at 24.5 Gb/s

Misc Commands

Check driver info with modinfo

root@host:~# modinfo mlx5_core
filename:       /lib/modules/5.15.0-67-generic/updates/dkms/mlx5_core.ko
alias:          auxiliary:mlx5_core.eth-rep
alias:          auxiliary:mlx5_core.eth
basedon:        Korg 6.0-rc4
version:        5.9-0.5.5
license:        Dual BSD/GPL
description:    Mellanox 5th generation network adapters (ConnectX series) core driver
author:         Eli Cohen <eli@mellanox.com>
...

Links

Forum discussion on host chaining: https://forums.developer.nvidia.com/t/how-to-configure-host-chaining-for-connectx-5-vpi/207154

-- Tom Cauduro - 2023-04-12

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	RingTopo.png	r1	manage	46.8 K	2023-04-12 - 13:56	TomCauduro

Topic revision: r2 - 2023-04-12 - TomCauduro

Information in this area is meant for use by CSCF staff and is not official documentation, but anybody who is interested is welcome to use it if they find it useful.

Other Webs

My links
- People
- CERAS
- WatForm
- Tetherless lab
- Ubuntu Main.HowTo
- eDocs
- RGG NE notes
- RGG
- CS infrastructure
- Grad images

Edit