TWiki
>
Main Web
>
UserList
>
HarshRoghelia
>
NetworkingBreakdown
(2022-12-01,
HarshRoghelia
)
(raw view)
E
dit
A
ttach
---+ Research Support Group Meeting Agenda ---++ Date: 2022-12-01 * Place: MS Teams ---++ Attendance * Invited: Lori Paniak, Tom Cauduro, Nathan Fish, Harsh Roghelia ---++ Previous Meeting * [[RSG20221103]] ---++ Agenda ---+++ Networking! Firewalls! ---++++ Interfaces: Mellanox/NVIDIA, Chelsio, Intel, Broadcom(driver support?), Realtek, Marvell, Rockport (unpowered hub connecting nodes. High-power optical only NICs, experimental, $4k per 100Gbit NIC) * [[https://docs.nvidia.com/networking/display/ConnectX6EN]] * [[https://ark.intel.com/content/www/us/en/ark.html#@PanelLabel189556]] * Infiniband adapters are generally called Host Channel Adapter (HCA) * Mellanox VPI adapters are dual mode: ethernet_infiniband * Realizing performance: CPU clocks, Q/UPI rates, NUMA effects (single-socket, single PCIe root complex-CPU thing) * root complexes: [[https://www.supermicro.org.cn/products/system/4U/4029/PCIe-Root-Architecture.cfm]], [[https://www.servethehome.com/single-root-or-dual-root-for-deep-learning-gpu-to-gpu-systems/]] * tools: apt install infiniband-diags ---++++ Switches: Mellanox, Barefoot/Intel/P4, Broadcom/Tomahawk, HP, Cisco. Locked media * [[https://www.nvidia.com/en-us/networking/ethernet-switching/]] * Same vendor end-to-end on networking - if possible * HP/Aruba, Cisco switches require branded transceivers/DAC - contain a PROM with ID * lossless ethernet: Mellanox flow control. No dropped packets, needed for RoCE * Mellanox SN2100 - 16-port 100Gbit * Mellanox SN2010 - 18-port 25Gbit + 4x 100Gbit * Mellanox SN2700C - 32-port 100Gbit ---++++ Protocols: RoCE(RDMA over Converged Ethernet), Infiniband (HPC, low-latency fabric), RDMA (Remote Direct Memory Access), DPDK (Data Plane Development Kit), hostchaining * RoCE /OFED: Generally, Linux in-tree drivers are sufficient. For RoCE performance and HCA firmware updates: [[https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/]] * sea-store in ceph utilizes DPDK. seastar * Infiniband requires a fabric manager: often on switch but opensm can run on one of the fabric servers * Beware of heterogeneity in networking firmware/software: switch and NIC/HCA firmware should be in sync * hostchaining: * apt install mstflint * mstflint q |grep HOST * mstconfig -d 51:00.0 set HOST_CHAINING_MODE=1 * reboot system Transport: Twisted pair, Direct-attach, SFP/SFP28, QSFP, QSFP28, (Q)SFP56, QSFP-DD, Active optics: LC-LC, MPO, SR/LR, single-mode/multi-mode OS2, OM3/4/5 * [[https://www.fs.com/c/optics-and-transceivers-9]] * [[https://www.nvidia.com/en-us/networking/interconnect/]] * Higher rate (switch) ports can be broken out with direct-attach cable: 100-> 4x25 * QSA (quad-single adapter) works for NIC and switch: QSFP(28) -> SFP(28) * Direct-attach: copper cable. Generally good to 5m (25Gbit), 3m for (100Gbit). Use for in-rack. Relatively inexpensive and robust so long as you mind bend radius * Fiber: between racks * Typical plug is LC-LC * Newer/future: MPO. Multiple strand connector * Fiber types: * Single-mode (OS2) 1310nm: Good for legacy install compatible with current speed. Downside: expensive transceivers. Range is O(km) * Multi-mode (OM3/4/5) 850nm: OM4/OM5 do 100Gbit up to 150m. OM5 is broadband (850-950 nm) then transceiver uses Shortwave Wavelength Division Multiplexing (SWDM) with 4 non-overlapping wavelengths * Bend radius (See Resources): [[https://www.nvidia.com/en-us/networking/ethernet/direct-attach-copper-cables/]] Netplan: basic config, VLANs, bridges, MTU ip cheat sheet: https://access.redhat.com/sites/default/files/attachments/rh_ip_command_cheatsheet_1214_jcs_print.pdf Firewall: Shorewall. proxyarp -- %USERSIG{LoriPaniak - 2022-11-26}% ---++ Comments %COMMENT%
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r1 - 2022-12-01
-
HarshRoghelia
Main
Welcome TWikiGuest
Register
Log in
Main Web
Main Web Home
Users
Groups
Offices
Changes
Changes detailed
Topic list
Search
TWiki Webs
CSEveryBody
AIMAS
CERAS
CF
CrySP
External
Faqtest
HCI
Himrod
ISG
Main
Multicore
Sandbox
TWiki
TestNewSandbox
TestWebS
UW
My links
People
CERAS
WatForm
Tetherless lab
Ubuntu HowTo
eDocs
RGG NE notes
RGG
CS infrastructure
Grad images
Edit
Copyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback