Research Support Group Meeting Agenda

Date: 2022-12-01

* Place: MS Teams

Attendance

* Invited: Lori Paniak, Tom Cauduro, Nathan Fish, Harsh Roghelia

Previous Meeting

* RSG20221103

Agenda

Networking! Firewalls!

Interfaces: Mellanox/NVIDIA, Chelsio, Intel, Broadcom(driver support?), Realtek, Marvell, Rockport (unpowered hub connecting nodes. High-power optical only NICs, experimental, $4k per 100Gbit NIC)

• https://docs.nvidia.com/networking/display/ConnectX6EN
• https://ark.intel.com/content/www/us/en/ark.html#@PanelLabel189556
• Infiniband adapters are generally called Host Channel Adapter (HCA)
• Mellanox VPI adapters are dual mode: ethernet_infiniband
• Realizing performance: CPU clocks, Q/UPI rates, NUMA effects (single-socket, single PCIe root complex-CPU thing)
• root complexes: https://www.supermicro.org.cn/products/system/4U/4029/PCIe-Root-Architecture.cfm, https://www.servethehome.com/single-root-or-dual-root-for-deep-learning-gpu-to-gpu-systems/
• tools: apt install infiniband-diags

Switches: Mellanox, Barefoot/Intel/P4, Broadcom/Tomahawk, HP, Cisco. Locked media

• https://www.nvidia.com/en-us/networking/ethernet-switching/
• Same vendor end-to-end on networking - if possible
• HP/Aruba, Cisco switches require branded transceivers/DAC - contain a PROM with ID
• lossless ethernet: Mellanox flow control. No dropped packets, needed for RoCE
• Mellanox SN2100 - 16-port 100Gbit
• Mellanox SN2010 - 18-port 25Gbit + 4x 100Gbit
• Mellanox SN2700C - 32-port 100Gbit

Protocols: RoCE(RDMA over Converged Ethernet), Infiniband (HPC, low-latency fabric), RDMA (Remote Direct Memory Access), DPDK (Data Plane Development Kit), hostchaining

• RoCE /OFED: Generally, Linux in-tree drivers are sufficient. For RoCE performance and HCA firmware updates: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/
• sea-store in ceph utilizes DPDK. seastar
• Infiniband requires a fabric manager: often on switch but opensm can run on one of the fabric servers
• Beware of heterogeneity in networking firmware/software: switch and NIC/HCA firmware should be in sync
• hostchaining:
  • apt install mstflint
  • mstflint q |grep HOST
  • mstconfig -d 51:00.0 set HOST_CHAINING_MODE=1
  • reboot system

Transport: Twisted pair, Direct-attach, SFP/SFP28, QSFP, QSFP28, (Q)SFP56, QSFP-DD, Active optics: LC-LC, MPO, SR/LR, single-mode/multi-mode OS2, OM3/4/5

• https://www.fs.com/c/optics-and-transceivers-9
• https://www.nvidia.com/en-us/networking/interconnect/
• Higher rate (switch) ports can be broken out with direct-attach cable: 100-> 4x25
• QSA (quad-single adapter) works for NIC and switch: QSFP(28) -> SFP(28)
• Direct-attach: copper cable. Generally good to 5m (25Gbit), 3m for (100Gbit). Use for in-rack. Relatively inexpensive and robust so long as you mind bend radius
• Fiber: between racks
  • Typical plug is LC-LC
  • Newer/future: MPO. Multiple strand connector
• Fiber types:
  • Single-mode (OS2) 1310nm: Good for legacy install compatible with current speed. Downside: expensive transceivers. Range is O(km)
  • Multi-mode (OM3/4/5) 850nm: OM4/OM5 do 100Gbit up to 150m. OM5 is broadband (850-950 nm) then transceiver uses Shortwave Wavelength Division Multiplexing (SWDM) with 4 non-overlapping wavelengths
• Bend radius (See Resources): https://www.nvidia.com/en-us/networking/ethernet/direct-attach-copper-cables/

Netplan: basic config, VLANs, bridges, MTU ip cheat sheet: https://access.redhat.com/sites/default/files/attachments/rh_ip_command_cheatsheet_1214_jcs_print.pdf

Firewall: Shorewall. proxyarp

-- Lori Paniak - 2022-11-26

Comments