UbuntuScripts < CF

TWiki>

CF Web>Windows>ImageDeployment>ImageDeploymentGhost>RIP>UbuntuScripts (2017-02-09, DanielAllen)

EditAttach

-- MikeGore - 20 Mar 2012

ubuntu-scripts

Location

\\ASIMOV\images\ubuntu-scripts
asimov:/coregroup1/images/ubuntu-scripts
FILES in this TWIKI are for reference - check ASIMOV for the latest updates

Description

Scripts to bring a base Ubuntu Linux install up to CSCF standards and join AD
Documentation for scripts in this directory:

Tools

common

common Defines common shell functions
Usage:
- source common * *Notes: non-system users are the set 200 to 2000 in UID
Argument Processing
- -d sets DEBUG=TRUE
- -y sets DEBUG=TRUE and ANSWER=Y
- -n sets DEBUG=TRUE and ANSWER=N
Environment Variables Set
- RELEASENO=`lsb_release -rs`
- export PATH=$PATH:/usr/local/bin:
- DEBUG
Functions:
- backup file - make a create file.orig if file.orig does not exist
- sed_file CMD file - edits a file using sed command CMD
- compare_ge arg1 arg2 - true if numeric value arg1 >= arg2
- compare_le arg1 arg2 - true if numeric value arg1 <= arg2
- compare_gt arg1 arg2 - true if numeric value arg1 > arg2
- compare_lt arg1 arg2 - true if numeric value arg1 < arg2
- compare_eq arg1 arg2 - true if numeric value arg1 = arg2
- pause_no - echo string return true if user answers N or empty
  - Always returns true without prompting if ANSWER variable is N
- pause_yes - echo string return true if user answers Y or empty
  - Always returns true without prompting if ANSWER variable is Y
- pause - echo string and wait for input
  - Always returns is DEBUG variable = TRUE
- uprompt message default
  - TRUE, USER_PROMPT is set to default value - empty user input
  - FALSE, USER_PROMPT is set to user input - non empty user input
- get_user userid - set shell vars below for given user
  - PUID - userid from getent password
  - GRID - groupid from getent group
  - PGID - groupid from getent password
  - PPASS - encrypted password
  - PHOME - Home directory from getent password
  - PSHELL - SHELL from getent passwd
  - GRPS = all non-system user groups from get_groups function
- get_groups userid - lists users groups on STDOUT - less any system groups
  - Example VAR=`get_groups`
- list_users - list all non system users on STDOUT
  - Example: *VAR=`list_users`
- make_sshpub
  - Create SSH keys for a User if they do not have one
  - Add their public key to their authorized_keys file - very useful in a cluster
- update_ssh - Create SSH Keys for all non system users if they do not already have one
- make_users - create a script on STDOUT that will replicate or update all non-system users
  - Example:
    - make-users >script
    - chmod 755 script
    - This script can now be copied to another machine
- mk_user_dirs dir
  - create a directory using the userids from list_users under dir
  - Example: mk_user_dirs /data
    - /data will have sub directories with the correct permissions for each non-system user returned by list_users

fix_scheduler

Updates the IO scheduler for all /dev/sd* disks to deadline
- File: fix_scheduler

grub-fix

purge/reinstall grub2 and modifies /etc/default/grub
- File: grub-fix

fixhost

Rename a linux host. Assumes machine is called scspcxxx
Usage: fixhost scspc577.cs changes the name to scspc577.cs

mntpart /dev/sd[a-z][1...]

mounts ands *chroot*s to a partition
- bind mounts /dev /proc /sys, etc virtual file systems in target mount point
  - Then calls mntproc
- creates need mount point automatically under /mnt
- File: mntpart

mntproc

Mounts any remaining mount point (if any),sets the environment and calls bash
called by mntpart
- File: mntproc

mnt_asimov

SSH mounts a number of key ASIMOV directories under ~/asimov
- File: mnt_asimov

mnt_edocs

SSH mounts the CORE edocs directory under ~/edocs
- File: mnt_edocs

mnt_core

SSH mounts users CS CORE account to ~/CS-CORE
- File: mnt_core

lc

UofW lc source and binary
used by update script: see UbuntuImageCreation

adsetup and LDAP

See https://www.cs.uwaterloo.ca/twiki/view/CF/UbuntuLdapMIke
- https://www.cs.uwaterloo.ca/twiki/view/CF/SettingUpADGradPCs

./scripts

backup - script that backs up a system to a backup drive

backup rsync script with error checking and reporting
\\ASIMOV\images\ubuntu-scripts\scripts
Location: asimov:/coregroup1/images/ubuntu-scripts/scripts
- File: backup

Installation and setup

File: README.txt
This document describes the step to:
- setup and test postfix email for reports
- install scripts
- partition/format a backup drive (if not yet done)
- mount backup drive via /etc/fstab
- setup cron task to automate backup

backup command line usage

USAGE: $0 source_dir  target_dir free_space email
   Backup system root "/" to backup drive
   source_dir = source directory to backup
   target_dir = backup directory on backup drive
   free_space  = free space in megabytes we reserve on backup drive
   email = fully qualified email id of user to send alerts to
   Example:
      ./backup  /  /backup  /root  10000 userid@cs.uwaterloo.ca
      This will backup / into /backup/root checking there is 10G free
      If there are any errors send email to userid@uwaterloo.ca

manual backup

Example script to manually run the backup
- File: manual

crontab backup example

example crontab entry for the backup script
File: crontab

testfree

tests a mounted partition to confirm it has enough free space
Example:

   if /scripts/testfree /mnt/backup -f10000
   then
     echo I am ok!
   fi

testfree source and Makefile

Makefile: Makefile
Source File: testfree.c
Note Makefile copies the testfree.c file from: /home/cscf-adm/src/testfree.c

Disk Error Reporting - SMART and MegaRaid

Updated: 16 Nov 2012 Mike Gore

Description: SMART reporting tools
Summary: These tools can report drive information such as drive serial numbers, modle numbers and drive status
Location:
- \\ASIMOV\images\ubuntu-scripts\disk-utils
- asimov:/coregroup1/images/ubuntu-scripts/disk-utils
Getting Scripts
- - rsync -a cscf-adm@asimov.uwaterloo.ca:/coregroup1/images/ubuntu-scripts/disk-utils .
URL: https://cs.uwaterloo.ca/twiki/edit/CF/UbuntuScripts

10.04 Notes

smartmontools_5.40-1ubuntu1_amd64.deb
- This can be used on Ubuntu 10.04 to provide MegaRaid support
- dpkg -i smartmontools_5.40-1ubuntu1_amd64.deb

SMART

SMART monitoring scripts

DRIVES - defines system dirves to be scanned and shell functions

You only have to change TWO lines at the start smart and DRIVES to make it work in your system

Here is start of the DRIVES file

Look at the DRIVES= and smart() lines

   # CONFIGURATION SECTION - define your host here
   # ASIMOV
   export DRIVES=`echo /dev/sd{a..g}`
   smart() { run smartctl -a -d ata $1; }

   # WHOOMP
   # export DRIVES=`echo /dev/sg{2..17}`
   # smart() { run smartctl -a -d scsi $1; }

   # M160
   # export DRIVES="0 1 `seq 3 64`"
   #               No drive on location 2
   # smart() { run smartctl -a -d megaraid,$1 /dev/sda; }

drive-health - pass/fail status of all of the drives
drive-inventory - inventroy and error reporting of all drives

SMART example report

   root@whoomp:~/utils# ./drive-inventory
   ============================================
   Drive: /dev/sg2
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38DAF
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg3
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39FLV
   SMART Health Status: HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE [asc=5d, ascq=10]
   ============================================
   Drive: /dev/sg4
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39CFJ
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg5
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38GDL
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg6
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39FPJ
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg7: megaraid perl script to check drive health
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS2ZPDC
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg8
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38DAL
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg9
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS375HT
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg10
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS395BK
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg11
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38G9H
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg12
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39CJ6
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg13
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39CS1
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg14
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39849
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg15
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS38ECV
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg16
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39D22
   SMART Health Status: OK
   ============================================
   Drive: /dev/sg17
   Device: ST31500341AS Version: CX1H
   Device type: disk
   Serial number: 9VS39DT9
   SMART Health Status: OK
   ============================================



   ============================================
   **************  ERROR REPORT **************
   ============================================
   Drive: /dev/sg3
   smartctl -a -d scsi /dev/sg3
   smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
   Home page is http://smartmontools.sourceforge.net/

   Device:          ST31500341AS     Version: CX1H
   Serial number:             9VS39FLV
   Device type: disk
   Transport protocol: SAS
   Local Time is: Fri Nov 16 15:53:37 2012 EST
   Device supports SMART and is Enabled
   Temperature Warning Disabled or Not Supported
   SMART Health Status: HARDWARE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE [asc=5d, ascq=10]

   Error Counter logging not supported
   Device does not support Self Test logging
   ============================================

MegaRaid

mega raid reporting tools LSI disk controller

MegaCli64
- mega raid 64bit binary
MegaCli
- mega raid 32bit binary

megaraid-health

perl program that scanns all drives and reports

Error code is set for success or any failures

   root@m160:~/utils/disk-utils# perl /usr/local/bin/megaraid-health
   OK - VirtualDrives=3, Degraded=0, Offline=0, PhysicalDevices=65, Disks=60,
   CriticalDisks=0, FailedDisks=0, MemoryCorrectableErrors=0,
   MemoryUncorrectableErrors=0

run_megaraid-health
- Script to run megaraid-health so you can view the result

Adaptec Raid

The aacraid package is not apart of Ubuntu but can be obtained from source
- http://ubuntuforums.org/showthread.php?t=1788356&page=2
CLI refrence: http://download.adaptec.com/pdfs/user_guides/cli_v6_50_18771_users_guide.pdf

Disk Cloning with bad blocks

Description: SMART reporting tools
Summary: These tools can report drive information such as drive serial numbers, modle numbers and drive status
Location:
- \\ASIMOV\images\ubuntu-scripts\disk-utils
- asimov:/coregroup1/images/ubuntu-scripts/disk-utils
Getting Scripts
- - rsync -a cscf-adm@asimov.uwaterloo.ca:/coregroup1/images/ubuntu-scripts/disk-utils .
URL: https://cs.uwaterloo.ca/twiki/edit/CF/UbuntuScripts

Files are also in the
See: DiskRecovery

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
ext	DRIVES	r1	manage	2.0 K	2012-11-16 - 16:30	MikeGore	DRIVE file
ext	Makefile	r1	manage	0.3 K	2012-03-21 - 11:56	MikeGore
txt	README.txt	r1	manage	2.8 K	2012-03-21 - 12:11	MikeGore
ext	backup	r1	manage	2.3 K	2012-03-21 - 11:32	MikeGore
ext	crontab	r1	manage	0.4 K	2012-03-21 - 12:00	MikeGore
ext	drive-health	r1	manage	0.2 K	2012-11-16 - 16:30	MikeGore	drives-health script
ext	drive-inventory	r1	manage	0.2 K	2012-11-16 - 16:30	MikeGore	drives-inventory script
ext	fix_scheduler	r1	manage	2.0 K	2012-03-21 - 11:27	MikeGore
ext	grub-fix	r1	manage	0.9 K	2012-03-21 - 11:28	MikeGore
ext	manual	r1	manage	0.2 K	2012-03-21 - 11:33	MikeGore
ext	megaraid-health	r1	manage	2.7 K	2012-11-16 - 16:31	MikeGore	megaraid perl script to check drive health
ext	mnt_asimov	r1	manage	3.0 K	2012-03-21 - 10:25	MikeGore
ext	mnt_core	r1	manage	0.4 K	2012-03-21 - 11:26	MikeGore
ext	mnt_edocs	r1	manage	2.5 K	2012-03-21 - 11:32	MikeGore
ext	mntpart	r1	manage	0.8 K	2012-03-21 - 11:26	MikeGore
ext	mntproc	r1	manage	0.2 K	2012-03-21 - 11:27	MikeGore
c	testfree.c	r1	manage	3.1 K	2012-03-21 - 11:59	MikeGore

Topic revision: r12 - 2017-02-09 - DanielAllen

Information in this area is meant for use by CSCF staff and is not official documentation, but anybody who is interested is welcome to use it if they find it useful.

Other Webs

My links
- People
- CERAS
- WatForm
- Tetherless lab
- Ubuntu Main.HowTo
- eDocs
- RGG NE notes
- RGG
- CS infrastructure
- Grad images

Edit