tml.cs, tml2.cs and tml3 hardware build notes and software configuration

Contacts

• These machines are owned by Yaoliang Yu
• Managed by Mike Gore

Remote Admin Access

• ssh cscf-adm@tml.cs.uwaterloo.ca (F17 password)
• ssh cscf-adm@tml2.cs.uwaterloo.ca (F17 password)
• ssh cscf-adm@tml3.cs.uwaterloo.ca (F19 password)

Local Admin Access

• tml.cs
  • cscf-adm (2017 password)
• tml2.cs
  • cscf-adm (2019 password)
• tml2.cs
  • cscf-adm (2019 password)

TML hardware inventory

• Motherboard: ASUS WS X299 SAGE
• Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz
• Memory: 64GB (4 x 16GB Corsair CMK32GX4M2A2666C16 2133MHZ)
• Disks: Samsung NVME 2TB SSD
• GPU GeForce RTX 2080 and GeForce GTX 1080

TML2 hardware inventory

• Motherboard: ASUS X399 Taichi
• CPU: AMD Ryzen Threadripper 2920X 12-Core Processor
• Memory: 48GB (3 x 16GB Corsair CMK32GX4M2A2666C16 2133MHZ)
• Disks: Samsung NVME 1TB SSD
• GPU GeForce RTX 2080

TML3 hardware inventory

• Motherboard: ASUS WS X299 SAGE
• CPU: Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz
• Memory: 64GB (4 x 16GB Corsair CMK32GX4M2A2666C16 2133MHZ)
• Disks: Samsung NVME 1TB SSD
• GPU
  • NVIDIA Corporation GV100 TITAN V
  • NVIDIA Corporation GV100 TITAN V

Notes on the original purchase of TML

• The main constraint in the design of this machine was to permit up to 4 liquid cooled GPU cards in one chassis.
• The biggest problem we faced is that most GPU cards are air cooled from the side - putting 4 every other slot would obstruct the fans
• I9 System with 64G ram (expandable to 128G)
• Mother board needs at least 7 slots and the chassis must have 8 slots (GPU cards are two slots wide) the
  • GPU cards plug into slots: 1,3,5,7
• Build is only cost effective if you plan to add the 4 GPU's - currently the CPU is 35% of the overall cost
• Overall this system runs quiet as all of the fans ar 120mm and because of the large radiators they do not need to run fast

Parts for TML

• GPU - liquid cooled
  • EVGA GeForce GTX 1080 Ti FTW3 Hybrid
  • https://www.amazon.ca/EVGA-GeForce-Hybrid-Gaming-Technology/dp/B074D7S8HR
  • Notes: This card came with the liquid cooler - however many of the EVGA GPU cards can been be upgraded with EVGA liquid cooler add ons
  • FYI: Any 11G ram GPU car is going to be very hard to find in stock
• Case - lots of room for liquid cooling radiators
  • Item#: N82E16811133338
  • Thermaltake Core X5 Tempered Glass Edition E-ATX Stackable Black Cube Chassis CA-1E8-00M1WN-02
  • https://www.newegg.ca/Product/Product.aspx?reviews=all&Item=N82E16811133338
  • Notes: This case is stack-able - is wide enough that the motherboard is mounted horizontally
  • I really liked working with this case for this configuration - there are a number of places to put liquid cooling radiators
• Motherboard - 7 slot, lots of PCIe lanes and two NVMe disk slots
  • MFG #: WS X299 SAGE - Asus WS X299 SAGE Workstation Motherboard
  • http://www.directdial.com/WS%20X299%20SAGE.html
• CPU - I9
  • MFG #: BX80673I97940X
  • Intel Core i9 i9-7940X Tetradeca-core (14 Core) 3.10 GHz Processor
  • http://www.directdial.com/BX80673I97940X.html
• CPU cooler - liquid cooled
  • MFG #: CW-9060032-WW
  • Corsair H115i PRO Cooling Fan/Radiator
  • http://www.directdial.com/CW-9060032-WW.html
• RAM - must be purchased in groups of 4
  • Corsair Vengeance LPX 32GB (2x16GB) DDR4 DRAM 2666MHz C16 Memory Kit
  • MFG #: CMK32GX4M2A2666C16
  • http://www.directdial.com/CMK32GX4M2A2666C16.html
• NVMe disk
  • MFG #: MZ-V7E2T0BW
  • Samsung 970 EVO MZ-V7E2T0BW 2 TB Internal Solid State Drive PCI Express - M.2 2280
  • http://www.directdial.com/MZ-V7E2T0BW.html
• Power Supply
  • MFG #: 120-G2-1600-X1
  • EVGA SuperNOVA 1600 G2 Power Supply
  • http://www.directdial.com/120-G2-1600-X1.html
  • For up to 4 GPUs at max workload

Software Documentation

• Note - I will be moving the GPU setup scrips to aother TWIKI has they are updated frequently - I push out the changes to all of the TML machines - and others with GPUs

Wed Feb 12 13:40:41 EST 2020 IMPORTANT UPDATE tml3 
We now have cuda 10, cuDNN 7.31, tensorflow, pytorch and keras installed
   The packages are highly interdependent on specific cuda versions chosen.
   So we must use anaconda to permit private python environments. 

I created an anaconda environment called "ml"  - for math learning
   FYI: tensorflow and pytorch use "ml" for their installation.

To use the "ml" environment:
    source "/home/gpu-setup"/install_env  - this sets search paths and library paths
    source activate ml         - makes sure that you are in the ml workspace!

I created a Linux system group called ml to permit sharing of code
    You run "/home/gpu-setup"/update_ml_users as root any time to all all users to the ml group
    Example ml group sharing: chgrp -R ml /home/share;  chmod -R g+w /home/share

The following directories belong to the "ml group and all their files have group write added to them
    /usr/local/cuda*
    /usr/local/anaconda3
    "/home/gpu-setup"/cudnn_samples_v7

Installation scripts located in "/home/gpu-setup" were run as follows:
  install_first      - installs all required Ubuntu packages
  install_anaconda   - creates - anaconda ml environment - reboot after this
Note: the following scripts can be run any time if the system gets broken
  install_cuda       - reboot after this
  install_cuDNN      - installs cuDNN
  install_pycuda     - uses ml environment
  install_tensorflow - uses anaconda ml environment
  install_pytorch    - uses anaconda ml environment
  install_keras      - uses anaconda ml environment
Tests:
  cd "/home/gpu-setup"
  ./test_cuda
  ./test_pycuda
  ./test_pytorch
  ./test_tensorflow
  ./benchmark_gpu

TML Pictures

• TML - with covers off power supply side view:
  

• TMP with covers off rear top view:
  

• TML covers off CPU side view:
  

Install scripts

• Note - I will be moving the GPU setup scrips to aother TWIKI has they are updated frequently - I push out the changes to all of the TML machines - and others with GPUs

• Note: These scripts are in constant development - please use the latest version of these scripts which can be found on cscf-adm@asimov.uwaterloo.ca:/cscf-adm/src/gpu-setup

• install_1st: initial install script - a few basic package installs

• install_2nd: install anaconda , create python environment "ml" for math learning, installed support packages

• install_cuda: install cuda 9.0 and drivers using nVidias site - removes any existing nvidia or cuda drivers

• install_cuDNN-7.05: install cuDNN 7.05

• install_tensorflow: install tensorflow

• install_pytorch: install pytorch

• install_env: source this file in your shell scripts to setup environment and libraries paths

• common_functions: support shell functions using in all scripts

• benchmark_gpu: Cuda benchmarks

• test_tensorflow: test tensorflow

• update_ml_users: add all users to the system group called ml