Research Support Group Meeting Agenda

Date: 2022-12-01

* Place: MS Teams


* Invited: Lori Paniak, Tom Cauduro, Nathan Fish, Harsh Roghelia

Previous Meeting

* RSG20221103


Networking! Firewalls!

Interfaces: Mellanox/NVIDIA, Chelsio, Intel, Broadcom(driver support?), Realtek, Marvell, Rockport (unpowered hub connecting nodes. High-power optical only NICs, experimental, $4k per 100Gbit NIC)

Switches: Mellanox, Barefoot/Intel/P4, Broadcom/Tomahawk, HP, Cisco. Locked media

  • Same vendor end-to-end on networking - if possible
  • HP/Aruba, Cisco switches require branded transceivers/DAC - contain a PROM with ID
  • lossless ethernet: Mellanox flow control. No dropped packets, needed for RoCE
  • Mellanox SN2100 - 16-port 100Gbit
  • Mellanox SN2010 - 18-port 25Gbit + 4x 100Gbit
  • Mellanox SN2700C - 32-port 100Gbit

Protocols: RoCE(RDMA over Converged Ethernet), Infiniband (HPC, low-latency fabric), RDMA (Remote Direct Memory Access), DPDK (Data Plane Development Kit), hostchaining

  • RoCE /OFED: Generally, Linux in-tree drivers are sufficient. For RoCE performance and HCA firmware updates:
  • sea-store in ceph utilizes DPDK. seastar
  • Infiniband requires a fabric manager: often on switch but opensm can run on one of the fabric servers
  • Beware of heterogeneity in networking firmware/software: switch and NIC/HCA firmware should be in sync
  • hostchaining:
    • apt install mstflint
    • mstflint q |grep HOST
    • mstconfig -d 51:00.0 set HOST_CHAINING_MODE=1
    • reboot system

Transport: Twisted pair, Direct-attach, SFP/SFP28, QSFP, QSFP28, (Q)SFP56, QSFP-DD, Active optics: LC-LC, MPO, SR/LR, single-mode/multi-mode OS2, OM3/4/5

  • Higher rate (switch) ports can be broken out with direct-attach cable: 100-> 4x25
  • QSA (quad-single adapter) works for NIC and switch: QSFP(28) -> SFP(28)
  • Direct-attach: copper cable. Generally good to 5m (25Gbit), 3m for (100Gbit). Use for in-rack. Relatively inexpensive and robust so long as you mind bend radius
  • Fiber: between racks
    • Typical plug is LC-LC
    • Newer/future: MPO. Multiple strand connector
  • Fiber types:
    • Single-mode (OS2) 1310nm: Good for legacy install compatible with current speed. Downside: expensive transceivers. Range is O(km)
    • Multi-mode (OM3/4/5) 850nm: OM4/OM5 do 100Gbit up to 150m. OM5 is broadband (850-950 nm) then transceiver uses Shortwave Wavelength Division Multiplexing (SWDM) with 4 non-overlapping wavelengths
  • Bend radius (See Resources):

Netplan: basic config, VLANs, bridges, MTU ip cheat sheet:

Firewall: Shorewall. proxyarp

-- Lori Paniak - 2022-11-26


Topic revision: r1 - 2022-12-01 - HarshRoghelia
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback