Title GPUInstallScripts

  • These scripts installs Cuda, Tensorflow, Keras, Pytorch, Pycuda and related packages
-- Mike Gore - 2020-02-14

Location of master repository

This Document was created on Tue Feb 18 13:07:48 EST 2020 by /home/gpu-setup/create_documents
  • asimov:/home/gpu-setup - which is symlinked to asimov.uwaterloo.ca:/cscf-adm/src/gpu-setup
    • scripts push_servers contains a script that updates all of these machines currently only working from cs-tech1.cs.uwaterloo.ca

Hosts using these scripts

  • ai-vector tml tml2 tml3 ming-gpu-1 ming-gpu-2 ming-gpu-3 ming-gpu-4 ming-gpu-p40 ming-gpu-v100 honeydew beaker cabernet

GPU Installation scripts /home/gpu-setup

  • We now have cuda 10, cuDNN 7.31, tensorflow, pycuda, pytorch and keras installed

Updating or Creating this document

* Note: the script /home/gpu-setup/create_documents contains the TWIKI source for this document * Please update /home/gpu-setup/create_documents and rerun it to update the notes

Anaconda environment ml

  • I created an anaconda environment called ml
    • "ml" stands for math learning
    • FYI: tensorflow, pycuda, pytorch and tensorflow use "ml" for their installation.
  • These python packages are very dependent on a specific cuda versions
    • WARNING to prevent destroying the system python invironment
      • You must always use anaconda "ml" environment*

Using the "ml" python environment

  • source /home/gpu-setup/install_env
    • This sets search paths and library paths
  • source activate ml
    • This makes sure that you are in the ml workspace!

Linux system group called ml

  • I created a new Linux system group "ml"
    • This group permits sharing code between users

Adding users to the ml system group

  • Run /home/gpu-setup/update_ml_users as root any time

Manually adding the ml group to files or directories

  • Example ml group sharing
    • chgrp -R ml /home/share
    • chmod -R g+w /home/share

directories alrwady added to the ml group

    • /usr/local/cuda
    • /usr/local/anaconda3
    • /home/gpu-setup/cudnn_samples_v7

Installation scripts located in /home/gpu-setup

  • Installed in the following order

* Aside: these next scripts can also be run anytime to fix broken installation

ml system group update script

  • update_ml_users: GPU scripts
    • This script adds all users to the ml system group

Tests

Cleanup Scripts

  • purge_anaconda
    • This file DELETES ALL ANACONDA USER ENVIRONMENTS
    • ONLY USE THIS IF YOU HAVE TO START OVER FROM SCRATCH

Comments

Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt CUDA_README.txt r2 r1 manage 3.4 K 2020-02-18 - 13:04 MikeGore gpu scripts
Texttxt README.txt r1 manage 0.9 K 2020-02-18 - 12:08 MikeGore GPU scripts
Unknown file formatext benchmark_gpu r1 manage 0.2 K 2020-02-18 - 12:07 MikeGore gpu scripts
Unknown file formatext common_functions r1 manage 86.3 K 2020-02-18 - 12:07 MikeGore gpu scripts
Unknown file formatext common_vars r1 manage 6.6 K 2020-02-18 - 12:07 MikeGore gpu scripts
Unknown file formatext create_documents r2 r1 manage 4.0 K 2020-02-18 - 13:04 MikeGore gpu scripts
Unknown file formatext install_anaconda r1 manage 1.0 K 2020-02-18 - 12:07 MikeGore gpu scripts
Unknown file formatext install_cuDNN r1 manage 1.4 K 2020-02-18 - 12:07 MikeGore gpu scripts
Unknown file formatext install_cuda r1 manage 3.2 K 2020-02-18 - 12:07 MikeGore gpu scripts
Unknown file formatext install_env r1 manage 1.8 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext install_first r1 manage 6.1 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext install_keras r1 manage 1.2 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext install_pycuda r1 manage 0.7 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext install_pytorch r1 manage 0.9 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext install_tensorflow r1 manage 2.1 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext purge_anaconda r1 manage 0.2 K 2020-02-18 - 12:08 MikeGore GPU Scripts
Unknown file formatext push_servers r1 manage 0.9 K 2020-02-18 - 13:05 MikeGore Copies these scripts to all managed system with these tools installed
Unknown file formatext test_cuda r1 manage 0.1 K 2020-02-18 - 12:08 MikeGore GPU scripts
Unknown file formatext test_pycuda r1 manage 1.7 K 2020-02-18 - 12:08 MikeGore GPU scripts
Unknown file formatext test_pytorch r1 manage 0.5 K 2020-02-18 - 12:08 MikeGore GPU scripts
Unknown file formatext test_tensorflow r1 manage 1.3 K 2020-02-18 - 12:08 MikeGore GPU scripts
Unknown file formatext update_ml_users r1 manage 0.3 K 2020-02-18 - 12:08 MikeGore GPU scripts
Unknown file formatext update_pytorch r1 manage 0.4 K 2020-02-18 - 12:08 MikeGore GPU scripts
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2020-02-18 - MikeGore
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback