-- MikeGore - 2015-04-23

Services overview

* Ubuntu 14.04LTS Linux OS

• TFTPboot server
  • /tftpboot/pxes - root folder
• SAMBA server
  • /tftpboot/pxes shares for imaging
• Node image repository
  • himrod.cs:/himrod.node - current image of working nodes on himrod
• Has private NAT firewalled network - 192.168.1/24 10.0.15/24
  • DHCP with local name server
  • PXE tftpboot functions
    • Boot Linux repair and imaging utilities or Linux network installers
  • DNS name server for NAT
  • SAMBA file shares
  • NFS file shares
  • APACHE Web service
    • DNSMASQ lease status

Himrod Setup and Configuration overview files

• Software under /cscf-adm/src provides a number of tools for setting up himrod
  • NETWORKING
    • /etc/network/interfaces network interfaces for system
      • These MUST be statically defined for our setup scripts to work
    • /cscf-adm/src/hosts/common_hosts.himrod*
      • The master Makefile in /cscf-adm/src copies this to /usr/local/bin/common_host
      • EXTIF="em4.529" defines the device name for the extrenal interface
      • INTIF="bond0" main internal network
      • INTNAT="TRUE" use NAT to access the outside
      • INT2IF="em3" secondary internal network
      • INT2NAT="TRUE" use NAT to access the outside
      • Other settings include SAMBA and university network ranges
  • TFTPBOOT PXE
    • /tftpboot/pxes is the PXE root folder
    • /cscf-adm/src/dnsmasq contains all configuration files used for
      • DHCP and DNS
      • /cscf-adm/src/dnsmasq/dnsmasq.common.himrod
        • Defines all himrod nodes and interfaces with simplified IP and MAC address notation
        • provides DNS and reverse DNS for all local networks
        • Example:
          • host himrod
            • himrod.localnet has address 192.168.1.1
          • host himrod.cs
            • himrod.cs has address 129.97.152.5
      • make processes /cscf-adm/src/dnsmasq/dnsmasq.common.himrod
        • Creates:
          • /etc/hosts - defines local addresses for all himrod nodes
          • /etc/resolv.conf using /cscf-adm/src/dnsmasq/resolv.template
          • /etc/dnsmasq.hosts - is an include file in /etc/dnsmasq.conf
            • Note: /etc/dnsmasq.conf ONLY includes dnsmasq.hosts - do not define anything else
      • TFTPBOOT

• • FIREWALL and NAT for internal private networks - optionally provides NAT
  • SAMBA shares - also used by imaging tools
  • NFS for live boot images

Initial Setup overview

Only Done once

• Edit /cscf-adm/src/dnsmasq/dnsmasq.common.himrod
  • This defines subnets and host/mac IP assignments
• Edit /etc/network/interfaced
  • define ALL networks statically
• Edit /cscf-adm/src/hosts/common_hosts.himrod
  • define interface names and roles
• Run /cscf-adm/src/install_packages
  • This installes required packages
• Run make all
  • runs install scripts
    • dnsmasq, samba, nfs setup sccf-adm accounts
  • runs update scripts
    • download pxe live images etc
  • installs NAT firewall

Setup Updates

• Run: make copy_scripts
  • installs common_functions, common_host and common_vars in:
    • /usr/local/bin
• Run make docs
  • Creates the TWIKI and HTML documenation from sources places them in:
    • /cscf-adm/src/twiki and /cscf-adm/src/html

Main configuration and setup directories

Networking

• /etc/network/interfaces - system network configuration
  • The setup scripts assume the network setting exist statically in order to work
• /cscf-adm/src/hosts/common_hosts.himrod defines interfaces and roles (ie is the interface to a NAT network)

Himrod Setup and Installation folders

• /cscf-adm/src
  • This contains all of the tools used to do the initial setup and configuration of himrod
• /cscf-adm/src - program sources
• /cscf-adm/src/dnsmasq - DNSMASQ / TFTPBOOT /DNS serices and configuration
• /cscf-adm/src/pxe - PXE boot files - a minimal working PXE tree with boot images
• /cscf-adm/src/hosts - host network configuration - defines interfaces used by all scripts*
• /cscf-adm/src/syslinux - Syslinux sources - used for PXE booting
• /cscf-adm/src/Idrac - iDrac scripts - Dell configuration and licencing scripts
• /cscf-adm/src/cluster - Cluster scripts
• /cscf-adm/src/twiki - Twiki documents - autogenerated
• /cscf-adm/src/html - HTML documents - autogenerated
• /cscf-adm/src.web - web based reporting utils

DNSMASQ /cscf-adm/src/dnsmasq

• dnsmasq is a single package that provides PXE BOOT, DNS, DHCP services

Makefile

• make is the only file run by and admin to update himrods dnsmasq settings
• compiles mkhosts.c
• runs install_dnsmasq
• Note: make can be run at ANY time to make sure all files are up to date

install_dnsmasq

• Only called from make ( ie. Makefile ) to install and configure dnsmasq using files defined in the following sections
• Stops nscd and dnsmasq
• updates /etc/resolv.conf using resolv.template files
• updates /etc/dnsmasq.hosts usings hsosts.template
• creates /etc/hosts , /etc/resolv.conf and /etc/dnsmasq.hosts
  • Updates /etc/dnsmasq.conf to include /etc/dnsmasq.hosts
• creates /etc/Networkmanager/dispatcher.d/00-dnsmaq trigger script to rewrite /etc/resolve.conf in case Network Manager is installed
  • This prevents other programs from messing up /etc/resolv.conf
• starts nscd and dnsmasq

DNSMASQ Config files

dnsmasq.common.himrod

• himrod specific configuration for dnsmasq
• Put all IP,MAC address and interface information for the cluster here

dnsmasq.common

• just a symlink to dnsmasq.common.himrod created by make install process

DNSMASQ Template Files

• * These files are preprocessed and added to the top of the automatically generated /etc/dnsmasq.hosts file
  • dnsmasq/dnsmasq.template.private
    • default DNSMASQ settings for ONE private network
  • dnsmasq/dnsmasq.template.private_two
    • default DNSMASQ settings for TWO private networks
  • dnsmasq/dnsmasq.template.private_three
    • default DNSMASQ settings for THREE private networks
  • dnsmasq/hosts.template
    • defines fixed records we always want in the /etc/hosts file
  • dnsmasq/resolv.template
    • defines fixed record we want added to the top of the /etc/resolv.conf file
    • Default external DNS servers - should be moved into common_host

DNSMASQ Compiled files

• To recreate these files:
  • cd /cscf-adm/src/dnsmasq
  • make

/etc/hosts

• Create from mkhosts and install_dnsmasq using dnsmasq.common.himrod and hosts.template

/etc/resolv.conf

• Create from mkhosts and install_dnsmasq using resolv.template

/etc/dnsmasq.hosts

• Create from mkhosts and install_dnsmasq using dnsmasq template files

dnsmasq.compiled.himrod

• The top level include for dnsmasq created by the make install process
• includes preporocessed dnsmasq.template files
• includes dnsmasq.common

/etc/NetworkManager/dispatcher.d/00-dnsmasq

• Network Manager trigger script - recreates /etc/resolv.conf if dnsmasq sees a status change
  • This overwrites any changes, Network Manager may do if the network port status changes, with correct information

DNSMASQ mkhosts and mkhosts.c

• preprocessor program that reads
  • dnsmasq.compiled.himrod and hosts.template
  • Writes /etc/dnsmasq.hosts - a file included from the otherwise totally blank /etc/dnsmasq.conf file
  • Write /etc/hosts
  • Creates DHCP entries, DNS records - both forward and reverse lookups for local networks

mkhosts Ducumenation

• What: mkhosts.c a preprocessor for dnsmasq configuration files
• The preprocessor handles creating reverse dns records and settings from
• a simpilfied syntax
• Syntax:
• ttl= time
• time is a ttl as per the dnsmasq man pages
• Example: ttl=1440m - set the default ttl to 1440 minutes until the next ttl keyword
• * subnet= subnet specify the subnet
• subnet is a subnet specification - LIMITATIONS mkhosts uses hard coded /24 network - first 24 bits
• Example: subnet=192.168.1 - this defines the subnet the same as 192.168.1.0 .. 192.168.1.255
• ip,*mac*,*cname* [,*cname*]...
• ip is the last digit of the subnet only
• mac is the MAC address of the interface you want to assign the address to
• cname a common separated list of cnames
• Example: 101,00:25:90:29:eb:ec,m160-1-local-comm,m160-1
• * Creates

•    *     dhcp-host=192.168.2.101,00:25:90:29:eb:ec,m160-1-local-comm,1440m
     *     ptr-record=101.2.168.192.in-addr.arpa.,"m160-1-local-comm"
     *     address=/m160-1-local-comm/192.168.2.101
     *     cname=m160-1,m160-1-local-comm
     *     

• prefix= We can redefine dhcp name prefix for reverse lookups
• iface= just puts a comment in the output file
• tag= defind tags to be appened to dhcp-host
• pxelinux= Define the pxelinux subnet - hosts under this section can get a private boot file
• Each mac address will simlink to the file pxelinux.cfg/nodes
• include= process an include file
• Example:
• ttl=1440m
• subnet=192.168.2
• 1,00:25:90:0e:5b:a2,m160-local-comm,m160-comm,m160
• 101,00:25:90:29:eb:ec,m160-1-local-comm,m160-1
• ...

BASH Scripts summary

• accounts - default cscf-adm and cscf-op encrypted passwords
• cleanup - Delete files under a directory DIR and children that are older the DAYS old
  • ./cleanup directory days
• common_functions
  • Common shell scripts used by most scripts under /cscf-adm/src
• common_vars
  • Common directory locations and definitions
• common_host
  • compiled into /usr/local/bin from /cscf-adm/src/hosts/common_hosts.himrod
• cscf-root-pub-key.txt
  • copy of CSCF root public key to be added to /root/.ssh/autorized_keys2 file
• distroy_and_clone
  • himrod specific scripit added to RIPimage
  • Partitions and formats all drives on a node - mounts /dev/sda1 on /mnt/sda1
  • Runs: rsync -a -x -H --numeric-ids homrod:/himrod.node/ /*

. * dnsmasq.sh - lists dnsmasq leases

• fetch_clonezilla - grabs lates clonezilla image and installes it in /tftpboot/pxes/clonezilla
  • see: /tftpboot/pxes/pxelinux.cfg/clonezilla_menu
• fetch_pxe
  • Capture existing PXE code and images for the Image Deployment System
  • Not used on himrod
• fetch_rip
  • Downloads lates version of RIP
• fetch_ubuntu_installers
  • Downloads Ubuntu PXE instalers for any iso images under /tftpboot/pxes/rawiso/Ubuntu
  • Updates /tftpboot/pxes/ubuntu-installer/menu.conf called from master pxelinux menu
• fetch_ubuntu_releases
  • Downloads ubuntu iso images to /tftpboot/pxes/rawiso/Ubuntu
  • Example: ./fetch_ubuntu_releases 12.04 14.04 beta*
• firewall * NAT firewall service for system
  • Uses /usr/local/bin/common_host and helper scripts common_vars and common_functions
• fixhost
  • change hostname and ssh keys for a host - update postfix
• fixperms
  • Update the permissions on files under a directory
  • Only update those files that have them wrong so that backups do not update files needlessly
• flush_cache
  • This script will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
• format_disks
  • Partition and Format disks on a himrod node to prepare for imaging
• grub_fix
  • Updates /etc/default/grub, purges then reinstalls grub
• install_admin_users
  • Add local Administrative and Samba users defined in $ADM_USERS - see common_host
• install_dell_utils
  • Install Dell OpenManage Ubuntu Repository - not used on himrod
• install-networker
  • Install Legato Networker for CSCF backup services
• install_packages
  • Install common Ubuntu packages required for the Image Deployment System
• install_pxe
  • Install the PXE code and images for the Image Deployment System
  • We only install install the files once from the pxe folder
  • We should consider merging updates FROM the pxe folder to the pxe folder or the reverse
  • Correctly we use fetch_pxe (this excludes files that can be regenerated)
  • make fetch_pxe will do this
• install_samba
  • setup cscf-adm and cscf-op local user
    • Prompts to create an iinitial samba password for each account
• install_syslinux
  • Install and compile syslinux (the default Ubuntu version is out of date)
  • Installs files in /tftpboot/pxes/syslinux
• install_ubuntu_scripts
  • not used on himrod
• IPMIview
  • Supermicro IPMI utility wrapper
  • Console access and IPMI settings
• lc.c and lc
  • UofW list catalog program
• list_samba_users
  • List current samba users
• Makefile
  • Master make file calls all install and update scripts
• mntpart
  • used to chroot mount a partition - used by recovery tools like RIP
• mntprox
  • bind mounts partitions into chrooted environment
• NODES
  • Defines all of the cluster nodes by name and ILOM name
• pxe2usb.sh
  • Converts himrod PXE boot enviornment into standalone USB KEY
• pxe_liveboot_nfs
  • Service to mount Live Boot images
• readiso
  • Read an CD or DVD and create an ISO image
• sh2twiki and sh2twiki.c
  • Creates TWIKI and HTML documenation form simplified Markup Language
• syslinux-6.02.tar.gz
  • Syslinux source
• testfree and testfree.c
  • Used to verfify a partition is mounted and has enough free space
• test_vars
  • Lists variables used by common_vars, common_host and common_functions
• update_fstab
  • Updates /etc/fstab on nodes to include NFS home directories
• update_rip
  • Apply local costomizations to local RIP images
• updates
  • Add additional repositories, NTP and SMTP configuration, created symlinks need for CS AD
  • Used on clean system after first install
• wakeup uses wake on lan feature to wakeup any host listed in /etc/dnsmaqsq.hosts

firewall script provides firewall and NAT services for a private network

• Notes:
  • If you need to add or limit existing service rules
  • You should only have to make changed to the functions:
  • local, and less likely; mail_firewall

check()

• What:
  • Must have a NAT defined or die
• Return:
  • exit 1 if no firewall is defined

init_firewall()

• What:
  • remove all firewall rules and kernel modules
  • sets state to ACCEPT or DROP
• Arguments:
  • allow - or - drop
  • allow = ACCEPT all traffic
  • drop = DROP all traffic

list()

• What:
  • List network interfaces

hosts_allow()

• What:
  • Modifies /etc/hosts.allow adding the private networks

local_rules()

• What:
  • Local default firewall rules for UW
• Notes:
  • Customize these to suit your needs

main_firewall()

• What:
  • Main firewall code - site independent firewall code
• Notes:
  • Calls local_rules

add_ssh_host()

• What:
  • Open an inbound ssh rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

del_ssh_host()

• What:
  • Close an inbound ssh rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

checkhost()

• What:
  • Check if a host has a firewall rule
• Arguments:
  • host
• Returns:
  • Display iptables result

openhost()

• What:
  • Open an inbound rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

blockhost()

• What:
  • CLose an inbound rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

install_service()

• What:
  • Install firewall script as a service
• Notes:
  • Sets the firewall state to allow all
• Returns:
  • Displays status

remove_service()

• What:
  • Remove firewall service
• Notes:
  • Sets the firewall state to allow all
• Returns:
  • Displays status

Main Code Section:

• Arguments:
  • start - start firewall
  • stop - stop firewall
  • restart - restart firewall
  • status - firewall status display
  • add_ssh_host - add inbound allow ssh rule for host
  • del_ssh_host - del inbound allow ssh rule for host
  • openhost - add inbound allow open rule for host
  • blockhost - add inbound block rule for host
  • checkhost - check host rule
  • install - install this script as a service
  • uninstall

C Programs

• basestr.c
  • :Match substring
  • Returns: return( strncmp(argv[1],argv[2],len) == 0 ? 0 : 1);
• fixname.c
  • Escape strings so they can be safely passed to a shell
  • Names may be quoted with ''s
• lc.c
  • University of Waterloo List Catalogue command
• sh2twiki.c
  • Convert BASH and C markup to HTML or TWIKI
• testfree.c
  • Usage: testfree device -v -f threshold(in Megabytes)

Automatic Documentation

• make docs will automatically extract BASH and C markups
  • Creates TWIKI or HTML Documentation from Markup
• Features
  • Uses half the markup required the doxygen
  • Understands SHELL Variables and Functions

Markup Rules

• Heading: Markup syntax notes
• Comments:
  • Comments start with // or #
  • Examples
  • # this is a comment
  • // this is a comment
• Tokens:
  • Tokens are any word following a comment that ends with ':'
  • Examples
  • # Notes:
  • // Arguments: userid result
  • Notes: the token and any following comments are part of the token
• Reserved:
  • The following tokens have special meaning
  • Title: Creates a title for the document.
  • This keyword and any unbroken chain of comments and unreserved
  • tokens are part of the title and will be listed in the index file.
  • Author: Document Author, also Authour works
  • Heading: Create a heading
• Variables:
  • The following code objects have special meaning
  • VAR=value Variable assignments
  • This variable if following a token will become a comment
  • # ======== This a a section break
  • // ======= This is a section break
• Blank_Lines:
  • Blank Comments lines end a token or grouping
  • Examples:
  • //
  • #
  • C enum are processed
  • enum {
  • value1,
  • value2
  • };
• Functions: are processed along with the contents by counting brace levels
• Note: ANY unbroken preceding group of tokens/comments are part of the
  • function documentation.
  • Functions names in the documentation are listed BEFORE the preceding tokens
• Examples:
  • // This is a custom printf command
  • // Arguments: format string, variable number of arguments
  • // Returns: void
  • void myprintf(char *str, ...)
  • {
  • printf code goes here ...
  • }
• Special_Keywords:
• Processed keywords that create documentation sections
  • pause message
  • pause_yes message
  • pause_no message
  • must be followed by if,then,else,fo - optionly end with a section
  • Notes:
  • This is used in the ASIMOV updates script to create TWIKI documentation for
  • That script. The compatibility is included here

PXE

• PXE BOOT /tftpboot/pxes
  • Config: /tftpboot/pxes/pxelinux.cfg/default
  • ISO boot images: /tftpboot/pxes/iso

/tftpboot/pxes/pxelinux.cfg/default

PXE / TFTPBOOT files and descriptions

/tftpboot/pxes

• /tftpboot/pxes - PXE root directory Add any boot images under this directory - see below for specific directories that are already defined

/tftpboot/pxes/pxelinux.0

• PXE bootstrap file from syslinux - all systems boot this file first

/tftpboot/pxes/pxelinux.cfg/

• Top level PXE menu files go here - defined in binary file pxelinux.0

/tftpboot/pxelinux.cfg/default

• Top level PXE boot menu

/tftpboot/pxelinux.cfg/default.conf

• default menu include used by al menus
  • defines vesa display mode and resolution, defaulyt timeouts and prompts
  • you should not ever have to change this

/tftpboot/pxelinux.cfg/UW.png

• default graphic background for menus

/tftpboot/pxelinux.cfg/clonezilla_menu

• Clonezilla PXE boot menu
  • Primary imaging tool

/tftpboot/pxelinux.cfg/acronis_menu

• Acronis imaging tools menu

/tftpboot/pxelinux.cfg/rip_menu

• Recover is Possible RIP boot images
• there are rip_menuNN.N menus for each version
  • We used this tool to image the cluster - to be replaced by clonezilla

/tftpboot/pxelinux.cfg/dos_menu

• not used on himrod

/tftpboot/pxelinux.cfg/diag_menu

• not used on himrod

/tftpboot/pxelinux.cfg/live_menu

• Live boot images of Ubuntu

/tftpboot/pxes/syslinux

• Contains syslinux ustilities used by the menu system

/tftpboot/pxes/utils/

• used by dos_menu and diag_menu - not used on himrod See pxelinux.cfg/default

/tftpboot/pxes/RIP/

• Recovery is Possible - locally updated version of RIP PXE images

/tftpboot/pxes/avast

• Avast antivirus components - added to RIP images

/tftpboot/pxes/bin

• bin files added to RIP images

/tftpboot/pxes/clonezilla

• Colonezilla PXE boot images
• bin files added to RIP images

/tftpboot/pxes/images

• General purpose images folder free to use for deplyment images

/tftpboot/pxes/iso

• Ubuntu live iso shares used for PXE/NFS live booting

/tftpboot/pxes/pxelinux.cfg/ghost.src

• Linux Ghost code with additions to work with RIP

/tftpboot/pxes/ununtu-installers

• Ubuntu network installers

IDrac Scripts

Reset node using iDrac

• /cscf-adm/src/IiDrac/reset_node ilom-NODE - where NODE is lised in the table above

Apply iDrac License Keys to machines

• ./allnodes_apply_idrac_keys
  • Uses service_tags.txt for list of nodes - and to match the license file
    • Assumes license files are in current directory - see extract_keys

Setup boot drive on nodes himrod-1 .. 23

• ./allnodes_setup_boot_hd
  • Calls setup_boot_hd with node name

Setup disks on nodes himrod-1 .. 23

• ./allnodes_setup_disks
  • Calls setup_disks with node name

Setup node disks via iDRac

• ./setup_boot_hd nodename [-r]
  • nodename is like ilom-himrod-3
  • -r implies reset node

Update all nodes firmware

• ./allnodes_update_firmware
  • Uses service_tags.txt for list of nodes
  • Calls update_firmware

Update node firmware

• ./update_firmware nodename R620|R720xd
  • nodename is like ilom-himrod-3

Extract Dell license keys

• ./extract_keys
  • Extracts the License.zip file , then renames the license files so that the service tag for each node is part of the file name.
    • Example: DE00000040318201-1DF2DZ1.xml

List iDrac firmware versions for all nodes

• ./get_idrac_versions
  • Uses service_tags.txt for list of nodes

List node disks

• ./list_disks nodename
  • Lists iDrac disk and controller names on a node
  • nodename is like ilom-himrod-3

Reboot node into PXE - onceupdate_rip

• ./reboot_once_pxe nodename
  • nodename is like ilom-himrod-3

Reset node

• ./reset_node nodename
  • nodename is like ilom-himrod-3

Reset nodes

• ./reset-nodes
  • Resets all nodes except himrod-1 and storage nodes*

iDrac Help on the web

• Dell Remote Access Controller - DRAC / iDRAC
  • http://en.community.dell.com/techcenter/systems-management/w/wiki/3204.dell-remote-access-controller-drac-idrac.aspx
• RACADM Command Line Interface for DRAC
  • http://en.community.dell.com/techcenter/systems-management/w/wiki/3205.racadm-command-line-interface-for-drac.aspx

iDrac Directories

/cscf-adm/src/iDrac

• iDrac-Firmware-R620
  • Firmware
• iDrac-Firmware-R720xd
  • Firmware
• R620
  • Documents for R620
• orig
  • Backup of license files in zip format
• docs
  • iDrac documents integrated-dell-remote-access-cntrllr-7-v1.30.30_User's Guide_en-us.pdf

IDrac files

• account iDrac access password
• license_portal_url.txt
  • URL and userid for Dell license portal (This is where we downloaded our iDrac keys)
• racadm_readme.txt
  • Summary of iDrac commands (most of them listed below)
• service_tags_license_names.txt
  • License file n Resets all nodes except himrod-1 and storage nodes*

ames with node names and hardware type R620 or R720xd

• service_tags.txt
  • node name, service tag, hardware type R620 or R720xd
• Service Tags ver 4-1-2-20140605.xlsx
  • file with node names and service tags

iDRac files on tftpboot server

• /tftpboot/pxes/firmware/R620IDRAC/firmimg.d7
  • Firmare for R620 and R720xd - the happen to be the SAME image

IDrac ILOM network configurration examples

• • Set NIC to ILOM4
    • racadm config -g cfgLAnNetworking -o cfgNicSelection 5
  • Get NIC
    • racadm getconfig -g cfgLanNetworking -o cfgNicSelection
  • Static config
    • racadm setniccfg -s 172.19.128.254 255.255.255.0 172.19.128.1
  • Get NIC info - will list the ILOM and which interface it ties to
    • racadm getniccfg
  • Enable IPMI over lan
    • racadm config -g cfgIpmiLan -o cfgIpmiLanEnable 1
  • DHCP
    • racadm setniccfg -d

Idrac Firmware commands

* See update_firmware script - we flash update from the tftpboot server

• -p : upload firmware image from the client (RACADM)
• -g : use TFTP download process for firmware update (use with -u)
• -u : perform firmware update operation (used with -p and -g)
• -a : specifies TFTP server IP address used for firmware image (used with -g)
• -d : specifies source path where firmware image exists (default is local)
• -f : use FTP download process for firmware update
• -s : query for current firmware update process status
• -r : rollback to the standby firmware
• --clearcfg : after firmware update, removes the previous idrac configuration

Firmware update examples

* See update_firmware script - we flash update from the tftpboot server

• Upload a firmware image from the client and start firmware update:
  • racadm fwupdate -p -u -d /tmp/images
• Upload firmware image from FTP server and start firmware update:
  • racadm fwupdate -f 192.168.0.10 test test -d firmimg.d7
• Upload firmware image from TFTP server and start firmware update:
  • racadm fwupdate -g -u -a 192.168.0.100 -d /tmp/images
• Query the current status of the firmware update process:
  • racadm fwupdate -s
• Rollback to the standby firmware:
  • racadm fwupdate -r
• Upload firmware image from TFTP server, start firmware update. After firmware update is complete, delete previous iDRAC configuration:
  • racadm fwupdate -g -u -a 192.168.0.100 -d /tmp/images --clearcfg

ipmitool networking

• See: ClusterToolsIPMITOOL

Imaging a node

Live boot images

• /cscf-adm/src/fetch_ubuntu_releases 14.04 will update all of the latest 14.04 releases
  • Any number of version numbers can be listed - they will all be updated
  • calls pxe_liveboot_nfs service to remount and export new images

PXE Ubuntu Installers

• /cscf-adm/src/fetch_ubuntu_installers - creates a PXE installer for all liveboot images
• This is only of use for an install from scratch

RIP PXE node imaging

• We used RIP PXE boot to image the original nodes on himrod - the suggested new was is to use clonezilla
• Use iDRac web console for node
  • firefox --no-remote ilom-himrod-1
  • login with root and 2013 cscf-adm passwd
• Network boot the node in pxe mode: reboot_once_pxe
• Choose RIPLinuX 13.7
  • RIPLinuX 13.7 64Bit with Kernel from himrod version vmlinuz...
• login root - no password
• setup network dhcpcd
• run clone script: distroy_and_clone
  • This reformats /dev/sda partitions and copies /himrod-node (from himrod) to / on the node
• cd /mnt/sda1/ubuntu-scripts
• chroot and bind required filesystems into /mnt/sda1:
  • ./mnntpart /dev/sda1 - ignore any bind mount errors - a few are for other versions of ubuntu
• verify networking, ie: ping www.yahoo.com
• fix grub
  • cd /ubuntu-scripts
  • ./grubfix
    • yes remove grub - we purge and reinstall
    • boot */dev/sda1

RIP updates or customizati"> Local RIP updates or customization

• /cscf-adm/src/update_rip 13.7 pxe

USAGE: ./update_rip version  type
   version is numeric
   type is grub,grub2,syslinux,pxe or all
Example: ./update_rip 13.7 grub2
   patches RIP13.7 directory with ghost
Example: ./update_rip all
   updates ALL (pxe,syslinux,grub2) boot images on himrod

• This script will also add cscf specific customizations to RIP
  • It also rewrites the /tftpboot/pxes/pxelinux.cfg/rip_menu and rip_menu13.7 etc

Clonezilla PXE boot

• /cscf-adm/src/fetch_clonezilla updates to the latest version automatically
  • /tftpboot/pxes/pxelinux.cfg/clonezilla is the PXE boot menu - does not have to change with most updates.
  • We have the latest version (as of 10 April 2015) of clonezilla on himrod
• Web site: http://clonezilla.org/
• Screen shots
  • http://clonezilla.org/screenshots/?in_path=/00_Clonezilla
• Features
  • Can image and restore both Windows and Linux systems
  • Recover/backup
  • PXE bootable
  • Multicast
  • Supports a wide range of file systems
  • Backup only parts of the disk that are used
• Limitations
  • Cannot restore to smaller disks or partitions