UbuntuScripts < CF

CF Web>Windows>ImageDeployment>ImageDeploymentGhost>RIP>UbuntuScripts (2017-02-09, DanielAllen) (raw view)
-- Main.MikeGore - 20 Mar 2012
---++ ubuntu-scripts

---+++ Location
   * *\\ASIMOV\images\ubuntu-scripts*
   * *asimov:/coregroup1/images/ubuntu-scripts*
   * *FILES in this TWIKI are for reference - check ASIMOV for the latest updates*

---++ Description
   * Scripts to bring a base Ubuntu Linux install up to CSCF standards and join AD
   * Documentation for scripts in this directory: 
      * CF.UbuntuImageCreation
      * CF.UbuntuLdapMIke
      * CF.SettingUpADGradPCs
%TOC%
---++ Other Imaging Related Links

*These links are related to the imaging process itself*
   * CF.RIP 
      * Recovery is Posible Linux/Windows repair disk
   * CF.CSApplicationServer 
      * CSCF Application Server Documentation
   * CF.CSApplicationServerScripts 
      * CSCF Application Server Scripts Documentation
   * CF.ImageDeploymentAcronis 
      * Acronis Deployment Documentation
   * CF.ImageDeploymentGhost 
      * Ghost Deployment Documentation
   * CF.PXEBootFilesExample 
      * PXE Boot Server setup example
   * CF.PXELiveBoot 
      * Ubuntu PXE Live Boot images - Automatic download and NFS Export update

---++ Tools
---++++ common
   * *common* Defines common shell functions
   * *Usage:* 
      * *source common* * *Notes: non-system users are the set 200 to 2000 in UID
   * *Argument Processing* 
      * *-d* sets *DEBUG=TRUE*
      * *-y* sets *DEBUG=TRUE* and *ANSWER=Y*
      * *-n* sets *DEBUG=TRUE* and *ANSWER=N*
   * *Environment Variables Set* 
      * *RELEASENO=`lsb_release -rs`*
      * *export PATH=$PATH:/usr/local/bin:*
      * *DEBUG*
   * *Functions:* 
      * *backup* file - make a create file.orig if file.orig does not exist
      * *sed_file* CMD file - edits a file using sed command CMD
      * *compare_ge arg1 arg2* - true if numeric value arg1 &gt;= arg2
      * *compare_le arg1 arg2* - true if numeric value arg1 &lt;= arg2
      * *compare_gt arg1 arg2* - true if numeric value arg1 &gt; arg2
      * *compare_lt arg1 arg2* - true if numeric value arg1 &lt; arg2
      * *compare_eq arg1 arg2* - true if numeric value arg1 = arg2
      * *pause_no* - echo string return true if user answers N or empty 
         * Always returns true without prompting if ANSWER variable is N
      * *pause_yes* - echo string return true if user answers Y or empty 
         * Always returns true without prompting if ANSWER variable is Y
      * *pause* - echo string and wait for input 
         * Always returns is DEBUG variable = TRUE
      * *uprompt message default* 
         * TRUE, USER_PROMPT is set to default value - empty user input
         * FALSE, USER_PROMPT is set to user input - non empty user input
      * *get_user* userid - set shell vars below for given user 
         * *PUID* - userid from getent password
         * *GRID* - groupid from getent group
         * *PGID* - groupid from getent password
         * *PPASS* - encrypted password
         * *PHOME* - Home directory from getent password
         * *PSHELL* - SHELL from getent passwd
         * *GRPS* = all non-system user groups from get_groups function
      * *get_groups* userid - lists users groups on STDOUT - less any system groups 
         * *Example VAR=`get_groups`*
      * *list_users* - list all non system users on STDOUT 
         * *Example: *VAR=`list_users`*
      * *make_sshpub* 
         * Create SSH keys for a User if they do not have one
         * Add their public key to their authorized_keys file - very useful in a cluster
      * *update_ssh* - Create SSH Keys for all non system users if they do not already have one
      * *make_users* - create a script on STDOUT that will replicate or update all non-system users 
         * *Example:* 
            * *make-users &gt;script*
            * *chmod 755 script*
            * This script can now be copied to another machine
      * *mk_user_dirs dir* 
         * create a directory using the userids from *list_users* under *dir*
         * *Example: mk_user_dirs /data* 
            * /data will have sub directories with the correct permissions for each non-system user returned by *list_users*

---+++ fix_scheduler
   * Updates the IO scheduler for all */dev/sd** disks to *deadline* 
      * File: [[%ATTACHURL%/fix_scheduler][fix_scheduler]]

---+++ grub-fix
   * purge/reinstall grub2 and modifies */etc/default/grub* 
      * File: [[%ATTACHURL%/grub-fix][grub-fix]]
---+++ *fixhost*
   * Rename a linux host. Assumes machine is called *scspcxxx*
   * Usage:<strong> fixhost scspc577.cs </strong>changes the name to *scspc577.cs*
---+++ mntpart /dev/sd[a-z][1...]
   * mounts ands *chroot*s to a partition 
      * bind mounts /dev /proc /sys, etc virtual file systems in target mount point 
         * Then calls *mntproc*
      * creates need mount point automatically under */mnt*
      * File: [[%ATTACHURL%/mntpart][mntpart]]

---+++ mntproc
   * Mounts any remaining mount point (if any),sets the environment and calls bash
   * called by *mntpart* 
      * File: [[%ATTACHURL%/mntproc][mntproc]]

---+++ mnt_asimov
   * SSH mounts a number of key ASIMOV directories under *~/asimov* 
      * File: [[%ATTACHURL%/mnt_asimov][mnt_asimov]]

---+++ mnt_edocs
   * SSH mounts the CORE edocs directory under *~/edocs* 
      * File: [[%ATTACHURL%/mnt_edocs][mnt_edocs]]
---+++ mnt_core
   * SSH mounts users CS CORE account to *~/CS-CORE* 
      * File: [[%ATTACHURL%/mnt_core][mnt_core]]

---+++ lc
   * UofW lc source and binary
   * used by update script: see CF.UbuntuImageCreation

---+++ adsetup and LDAP
   * See https://www.cs.uwaterloo.ca/twiki/view/CF/UbuntuLdapMIke 
      * https://www.cs.uwaterloo.ca/twiki/view/CF/SettingUpADGradPCs

---++ ./scripts

---+++ backup - script that backs up a system to a backup drive
   * backup rsync script with error checking and reporting
   * *\\ASIMOV\images\ubuntu-scripts\scripts*
   * Location: *asimov:/coregroup1/images/ubuntu-scripts/scripts* 
      * File: [[%ATTACHURL%/backup][backup]]

---+++ Installation and setup
   * File: [[%ATTACHURL%/README.txt][README.txt]]
   * This document describes the step to: 
      * setup and test *postfix* email for reports
      * install scripts
      * partition/format a backup drive (if not yet done)
      * mount backup drive via */etc/fstab*
      * setup cron task to automate backup

---++++ backup command line usage
<verbatim>
USAGE: $0 source_dir  target_dir free_space email
   Backup system root "/" to backup drive
   source_dir = source directory to backup
   target_dir = backup directory on backup drive
   free_space  = free space in megabytes we reserve on backup drive
   email = fully qualified email id of user to send alerts to
   Example:
      ./backup  /  /backup  /root  10000 userid@cs.uwaterloo.ca
      This will backup / into /backup/root checking there is 10G free
      If there are any errors send email to userid@uwaterloo.ca
</verbatim>

---++++ manual backup
   * Example script to manually run the backup 
      * File: [[%ATTACHURL%/manual][manual]]

---++++ crontab backup example
   * example *crontab* entry for the *backup* script
   * File: [[%ATTACHURL%/crontab][crontab]]

---++++ testfree
   * tests a mounted partition to confirm it has enough free space
   * Example:
<verbatim>
   if /scripts/testfree /mnt/backup -f10000
   then
     echo I am ok!
   fi
</verbatim>

---++++ testfree source and Makefile
   * Makefile: [[%ATTACHURL%/Makefile][Makefile]]
   * Source File: [[%ATTACHURL%/testfree.c][testfree.c]]
   * Note Makefile copies the testfree.c file from: */home/cscf-adm/src/testfree.c*

---++ Disk Error Reporting - SMART and CFPrivate.MegaRaid

*Updated: 16 Nov 2012 Mike Gore*
   * *Description: SMART reporting tools*
   * *Summary: These tools can report drive information such as drive serial numbers, modle numbers and drive status*
   * *Location:* 
      * *\\ASIMOV\images\ubuntu-scripts\disk-utils*
      * *asimov:/coregroup1/images/ubuntu-scripts/disk-utils*
   * *Getting Scripts* 
      * 
         * *rsync -a cscf-adm@asimov.uwaterloo.ca:/coregroup1/images/ubuntu-scripts/disk-utils .*
   * *URL:* https://cs.uwaterloo.ca/twiki/edit/CF/UbuntuScripts
---+++ 10.04 Notes
   * *smartmontools_5.40-1ubuntu1_amd64.deb* 
      * This can be used on Ubuntu 10.04 to provide CFPrivate.MegaRaid support
      * *dpkg -i smartmontools_5.40-1ubuntu1_amd64.deb*

---+++ SMART

*SMART monitoring scripts*
   * *[[%ATTACHURL%/DRIVES][DRIVES]]* - defines system dirves to be scanned and shell functions
   * You only have to change TWO lines at the start *smart* and *[[%ATTACHURL%/DRIVES][DRIVES]]* to make it work in your system 
      * Here is start of the [[%ATTACHURL%/DRIVES][DRIVES]] file 
         * Look at the DRIVES= and smart() lines <verbatim>
   # CONFIGURATION SECTION - define your host here
   # ASIMOV
   export DRIVES=`echo /dev/sd{a..g}`
   smart() { run smartctl -a -d ata $1; }

   # WHOOMP
   # export DRIVES=`echo /dev/sg{2..17}`
   # smart() { run smartctl -a -d scsi $1; }

   # M160
   # export DRIVES="0 1 `seq 3 64`"
   #               No drive on location 2
   # smart() { run smartctl -a -d megaraid,$1 /dev/sda; }
   </verbatim>
   * *[[%ATTACHURL%/drive-health][drive-health]]* - pass/fail status of all of the drives
   * *[[%ATTACHURL%/drive-inventory][drive-inventory]]* - inventroy and error reporting of all drives
---++++ SMART example report
<verbatim>
   root@whoomp:~/utils# ./drive-inventory
   ============================================
   Drive: /dev/sg2
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38DAF
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg3
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39FLV
   SMART Health Status: HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE [asc=5d, ascq=10]
   ============================================
   Drive: /dev/sg4
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39CFJ
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg5
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38GDL
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg6
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39FPJ
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg7: megaraid perl script to check drive health
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS2ZPDC
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg8
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38DAL
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg9
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS375HT
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg10
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS395BK
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg11
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38G9H
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg12
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39CJ6
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg13
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39CS1
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg14
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39849
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg15
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38ECV
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg16
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39D22
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg17
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39DT9
   SMART Health Status: OK
   ============================================



   ============================================
   **************  ERROR REPORT **************
   ============================================
   Drive: /dev/sg3
   smartctl -a -d scsi /dev/sg3
   smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
   Home page is http://smartmontools.sourceforge.net/

   Device:          ST31500341AS     Version: CX1H
   Serial number:             9VS39FLV
   Device type: disk
   Transport protocol: SAS
   Local Time is: Fri Nov 16 15:53:37 2012 EST
   Device supports SMART and is Enabled
   Temperature Warning Disabled or Not Supported
   SMART Health Status: HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE [asc=5d, ascq=10]

   Error Counter logging not supported
   Device does not support Self Test logging
   ============================================
   </verbatim>
---+++ CFPrivate.MegaRaid

*mega raid reporting tools LSI disk controller*
   * *MegaCli64* 
      * mega raid 64bit binary
   * *MegaCli* 
      * mega raid 32bit binary
   * *[[%ATTACHURL%/megaraid-health][megaraid-health]]* 
      * perl program that scanns all drives and reports
      * Error code is set for success or any failures <verbatim>
   root@m160:~/utils/disk-utils# perl /usr/local/bin/megaraid-health
   OK - VirtualDrives=3, Degraded=0, Offline=0, PhysicalDevices=65, Disks=60,
   CriticalDisks=0, FailedDisks=0, MemoryCorrectableErrors=0,
   MemoryUncorrectableErrors=0
   </verbatim>
   * *run_megaraid-health* 
      * Script to run megaraid-health so you can view the result

---+++ Adaptec Raid
   * The *aacraid* package is not apart of Ubuntu but can be obtained from source 
      * http://ubuntuforums.org/showthread.php?t=1788356&page=2
   * CLI refrence: http://download.adaptec.com/pdfs/user_guides/cli_v6_50_18771_users_guide.pdf

---+++ Disk Cloning with bad blocks
   * *Description: SMART reporting tools*
   * *Summary: These tools can report drive information such as drive serial numbers, modle numbers and drive status*
   * *Location:* 
      * *\\ASIMOV\images\ubuntu-scripts\disk-utils*
      * *asimov:/coregroup1/images/ubuntu-scripts/disk-utils*
   * *Getting Scripts* 
      * 
         * *rsync -a cscf-adm@asimov.uwaterloo.ca:/coregroup1/images/ubuntu-scripts/disk-utils .*
   * *URL:* https://cs.uwaterloo.ca/twiki/edit/CF/UbuntuScripts

   * Files are also in the
   * See: CF.DiskRecovery
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
ext	DRIVES	r1	manage	2.0 K	2012-11-16 - 16:30	MikeGore	DRIVE file
ext	Makefile	r1	manage	0.3 K	2012-03-21 - 11:56	MikeGore
txt	README.txt	r1	manage	2.8 K	2012-03-21 - 12:11	MikeGore
ext	backup	r1	manage	2.3 K	2012-03-21 - 11:32	MikeGore
ext	crontab	r1	manage	0.4 K	2012-03-21 - 12:00	MikeGore
ext	drive-health	r1	manage	0.2 K	2012-11-16 - 16:30	MikeGore	drives-health script
ext	drive-inventory	r1	manage	0.2 K	2012-11-16 - 16:30	MikeGore	drives-inventory script
ext	fix_scheduler	r1	manage	2.0 K	2012-03-21 - 11:27	MikeGore
ext	grub-fix	r1	manage	0.9 K	2012-03-21 - 11:28	MikeGore
ext	manual	r1	manage	0.2 K	2012-03-21 - 11:33	MikeGore
ext	megaraid-health	r1	manage	2.7 K	2012-11-16 - 16:31	MikeGore	megaraid perl script to check drive health
ext	mnt_asimov	r1	manage	3.0 K	2012-03-21 - 10:25	MikeGore
ext	mnt_core	r1	manage	0.4 K	2012-03-21 - 11:26	MikeGore
ext	mnt_edocs	r1	manage	2.5 K	2012-03-21 - 11:32	MikeGore
ext	mntpart	r1	manage	0.8 K	2012-03-21 - 11:26	MikeGore
ext	mntproc	r1	manage	0.2 K	2012-03-21 - 11:27	MikeGore
c	testfree.c	r1	manage	3.1 K	2012-03-21 - 11:59	MikeGore
Topic revision: r12 - 2017-02-09 - DanielAllen
Information in this area is meant for use by CSCF staff and is not official documentation, but anybody who is interested is welcome to use it if they find it useful.
Other Webs
My links
- People
- CERAS
- WatForm
- Tetherless lab
- Ubuntu Main.HowTo
- eDocs
- RGG NE notes
- RGG
- CS infrastructure
- Grad images
Edit