m160 Bioinformatics Research Cluster

• This document describes Ming Li's CCF research cluster m160.cs

• See: https://www.cs.uwaterloo.ca/cscf/internal/edocs/machines/m160.cs/

EDOCS

• See: https://www.cs.uwaterloo.ca/cscf/internal/edocs/machines/m160.cs/

Quotas

• See UbuntuQuota System has quota setup but not used yet

Basic Health Test

• check-nodes
  • ping test of nodes
• check-ssh-nodes
  • SSH test of nodes
• check-ilom-nodes
  • ping test of node iloms

List Drive Serial Numbers using smartmontools

• /cscf-adm/src/disk-utils
• These scripts are presently not working

Syncing Packages between nodes

• Uses m160-1 as a reference node and duplicates all of its packages on each node
• Script: sync-packages

Syncing userid or password updates between nodes

• Provides: Sync account entries,passwords and SSH keys across nodes
  • Adds user SSH keys to their .ssh/authorized_keys file
• Script: sync-users

Remounting nodes

• fix-mount - run this on m160 as root Used if the head node restarts

Sync user accounts and ssh keys

• sync-users
  • syncs user accounts SSH keys and various scripts to all nodes

Rebooting Nodes

• reboot-nodes

DNSMASQ - PXE BOOT, DHCP, DNS

• 24 Feb 2022 - we switched to the updated PXE,BOOT/DNS/DHCP setup
  • See: https://cs.uwaterloo.ca/twiki/view/CF/ClusterToolsDNSMASQ
  • See: https://cs.uwaterloo.ca/twiki/view/CF/DNSMASQ
• See /cscf-adm/src/dnsmasq
• Provides: PXE BOOT, DNS, DHCP
• Updating DHCP and DNS information for nodes
  • cd /cscf-adm/src/dnsmasq
  • vi /cscf-adm/src/hosts/m160/dnsmasq.common
  • Restart: make
• PXE BOOT /tftpboot/pxes
  • Config: /tftpboot/pxes/pxelinux.cfg/default
  • Mounted ISO boot images: /tftpboot/pxes/iso
  • Clonezilla Images: /tftpboot/pxes/images
    • make_clonezilla rebuilds and mounts any NODE images

Firewall NAT

• 24 Feb 2022 - we switched to the updated PXE,BOOT/DNS/DHCP setup

* This critical script allows the nodes to talk to the outside world and limits access to the head node

• Startup script: /etc/init.d/firewall start|stop|status|restart|install
• Support scripts:
  • /usr/local/bin/common_host, /usr/local/bin/common_vars,/usr/local/bin/common_functions

Imaging Nodes

• See Remote Management
• PXE BOOT the node you want to Image by powering on or resetting the node
  • Press F12 during BIOS boot phase when you first see the BIOS screen - keep pressing the F12 until
    • - or - from m160
  • *ipmipxeboot lom-m160-NN (NN = 1..16)
    • Run this on m160
    • (cscf-adm, 2020 password when prompted)
  • ipmipoweron or ipmireset lom-m160-NN (NN = 1 .. 16)
    • Run this on m160
    • (cscf-adm, 2020 password when prompted)
• Pick Clonezilla

Image Archive Locations and Access

• tftpboot/pxes/images/m160-16-2021-05-18-04-img
  • Userid: cscf-adm - 2020 password in safe

Networking

• Notes: Purchased by Ming Li as part of the cluster
  • Networking is provide by two HP ProCurve 2910al-24G Ethernet Switches and a Dlink switch
• https://istns.uwaterloo.ca/ona/editdevice.php?ipname=cs-sw-mc-3015-cs2d
• https://istns.uwaterloo.ca/ona/editdevice.php?ipname=cs-sw-mc-3015-cs2e
• External network: vlan 129.97.2
  • Outside Feed: https://istns.uwaterloo.ca/ona/editport.php?ipname=cs-sw-mc-3015-cs2d&port=A2
• Local communication only: vlan 500 - 192.168.2.
  • cs-sw-mc-3015-cs2d Feeds all nodes and storage server
• NFS traffic only: vlan 501 - 10.0.2.
  • cs-sw-mc-3015-cs2e Feeds all nodes and storage server

IPMI View utility - Linux

• Provides: remote console and power managment of the cluster
• Notes: See IPMI for Ubuntu Linux utility
• Head Node: lom-m160.cs
• Nodes: lom-m160-N.cs
• Access: cscf-adm - password in safe - 2020
• Documentation: TWIKI page IPMI
• Start IPMI View utility
• All LOM interfaces live on network 172.19.96
  • Start IPMI View Search* using addresses 172.19.96.227 to 172.19.96.246
  • Save the LOM interfaces the utility finds
  • OK to exit
• Double click on the node you wish to manage - on left hand side under IPMI Domain
• Login: cscf-adm - password in safe - 2020
• Open KVM Console
• Open the "Soft Keyboard* so you can send special characters that your local OS may intercept

MegaCli Utility

Not working on 20.04LTS

• Reference:
  • https://twiki.cern.ch/twiki/bin/view/FIOgroup/DiskRefPerc
  • http://tools.rapidsoft.de/perc/perc-cheat-sheet.html
  • http://pookey.co.uk/wordpress/archives/46-dell-perc-6i-and-raid-monitoring
• /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL
• Rebuild progress
  • ./opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv \[32:1\] -aALL
• Cheat Sheet
  • http://tools.rapidsoft.de/perc/perc-cheat-sheet.pdf

Hardware and Inventory Section

Not working on 20.04LTS

• MegaCli64 -PDList -aALL

List All drive serial Numbers

Not working on 20.04LTS

• MegaCli64 -PDList -aALL | grep "Inquiry Data"

Updated 3 Nov 2011

root@m160:~/utils# MegaCli64 -PDList -aALL | grep "Inquiry Data"
   Inquiry Data: SEAGATE ST32000444SS    00069WM33GWT            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33KW5            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33LG7            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33CX1            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33GSG            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34F8D            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33G6P            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33LY8            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33M19            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33L2C            
   Inquiry Data: SEAGATE ST32000444SS    00069WM345BT            
   Inquiry Data: SEAGATE ST32000444SS    00069WM31XJP            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34FS4            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33KXB            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33KTK            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33LN4            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34DNT            
   Inquiry Data: SEAGATE ST32000444SS    00069WM31W2Y            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34FGR            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33LHQ            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34AAN            
   Inquiry Data: SEAGATE ST32000444SS    00069WM345E6            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34BEK            
   Inquiry Data: SEAGATE ST32000444SS    00069WM345FX            
   Inquiry Data: SEAGATE ST32000444SS    00069WM347EN            
   Inquiry Data: SEAGATE ST32000444SS    00069WM31Y3G            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34AWW            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33E8F            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34C3J            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33LWS            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34H9D            
   Inquiry Data: SEAGATE ST32000444SS    00069WM2XK5C            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33D50            
   Inquiry Data: SEAGATE ST32000444SS    00069WM34K29            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33LXA            
   Inquiry Data: SEAGATE ST32000444SS    00069WM33CH4            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P068XV            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P06ZSJ            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P05DS9            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P073CP            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P06EV3            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P06ZTV            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P06ZTC            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P05DLA            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P072H2            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P06EVR            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P04Z2Z            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P06JDV            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P067YK            
   Inquiry Data: SEAGATE ST2000NM0001    0001Z1P0600E        
   

PO with parts summary

* quote_1510258_4-1.pdf: PO with parts summary

APC SUA3000XLT UPS

Note installed presently after move to MC3015

• apt-get install apcupsd
• APC Smart-UPS XL 3000VA 208V Tower/Rack Convertible
  • http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=SUA3000XLT
• Inventory CS006523 APC UPS
• Inventory CS006524 APC UPS

AP5017 APC LCD/KVM

• APC AP5017 Rackmount LCD
  • http://www.apc.com/resource/include/techspec_index.cfm?base_sku=AP5017
• Inventory CS006521 APC KVM

Hardware and Documents

• X8DTN+-F Motherboard
  • http://www.supermicro.com/products/motherboard/QPI/5500/X8DTN_.cfm
• SC847E26-R1400UB Storage Chassis
  • http://www.supermicro.com/products/chassis/4U/847/SC847E26-R1400U.cfm
• AOC-USAS2LP-H4iR Storage Chassis Raid Controller
  • Description: http://www.supermicro.com/products/accessories/addon/AOC-USAS2LP-H8iR.cfm *Drivers and Docs: http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/6gb_s_value_line/sas9260-8i/index.html
  • Users Guide http://www.lsi.com/DistributionSystem/User/AssetMgr.aspx?asset=55129
  • Mega Raid for Ubuntu 10.04 http://www.lsi.com/lookup/License.aspx?url=/DistributionSystem/User/AssetMgr.aspx?asset=56645&prodName=MegaRAID%20SAS%209260-8i%20&subType=Driver&locale=EN
• Intel® Xeon® Processor X5680 (12M Cache, 3.33 GHz, 6.40 GT/s Intel® QPI)
  • http://ark.intel.com/Product.aspx?id=47916
• Seagate Constellation™ ES SAS 6Gb/s 2-TB Hard Drive
  • http://www.seagate.com/ww/v/index.jsp?name=st32000444ss-constellation-es-sas-6gbs-2-tb-hd&vgnextoid=f6dc38661f2a4210VgnVCM1000001a48090aRCRD&vgnextchannel=f424072516d8c010VgnVCM100000dd04090aRCRD&locale=en-US&reqPage=Support

Switches

• MC3015 cs2d
• MC3015 cs2e

Compute Nodes - Supermicro 6016TT-TF Twin 1U Intel Xeon 5600/5500 Series

• Server Super X8DTT-F + Super X8DTT-F

Hardware

• http://www.supermicro.com/products/system/1U/6016/SYS-6016TT-TF.cfm

Inventory and Access and LOM

• Inventory CS006517 Compute Node 1
  • Name: m160-1
  • IPMI: lom-m160-1.cs
• Inventory CS006513 Compute Node 2
  • Name: m160-2
  • IPMI: lom-m160-2.cs
• Inventory CS006518 Compute Node 3
  • Name: m160-3
  • IPMI: lom-m160-3.cs
• Inventory CS006514 Compute Node 4
  • Name: m160-4
  • IPMI: lom-m160-4.cs
• Inventory CS006519 Compute Node 5
  • Name: m160-5
  • IPMI: lom-m160-5.cs
• CS006515 Compute Node 6
  • Name: m160-6
  • IPMI: lom-m160-6.cs
• Inventory CS006520 Compute Node 7
  • Name: m160-7
  • IPMI: lom-m160-7.cs
• Inventory CS006516 Compute Node 8
  • Name: m160-8
  • IPMI: lom-m160-8.cs

Hardware and Documents

• http://www.supermicro.com/products/system/1U/6016/SYS-6016TT-TF.cfm

HP ProCurve 2910al-24G Ethernet Switch

• Inventory CS006510, HP ProCurve Switch, cs-sw-mc-3015-cs2d
• ONA: https://istns.uwaterloo.ca/ona/editdevice.php?ipname=cs-sw-mc-3015-cs2d
• Inventory CS006511, HP ProCurve Switch, cs-sw-mc-3015-cs2e
• ONA: https://istns.uwaterloo.ca/ona/editdevice.php?ipname=cs-sw-mc-3015-cs2e

Hardware and Documents

• HP E2910-24G al Switch (J9145A)
  • http://h10010.www1.hp.com/wwpc/uk/en/sm/WF06b/12883-12883-4172267-4172278-4172278-3901671-3901673.html
• HP 2-port 10GbE SFP+ al Module
  • http://h30094.www3.hp.com/product.asp?sku=3981098&mfg_part=J9008A&pagemode=ca
• HP X242 SFP+ SFP+ 3m Direct Attach Cable
  • http://h30094.www3.hp.com/product/sku/4024413/mfg_partno/J9283B
• HP X132 10G SFP+ LC LR Transceiver
  • http://h30094.www3.hp.com/product/sku/3976335/mfg_partno/J9151A

DES-1024D 24-port 10/100 Desktop/Rackmount Switch

• http://www.dlink.ca/products/?pid=75
• Inventory
  • CS006512 D-Link Switch

• Summary.ods: MAC address inventory for switches and m160 cluster ports