-- MikeGore - 2015-04-23

Services overview

* Ubuntu 14.04LTS Linux OS

• TFTPboot server
  • /tftpboot/pxes - root folder
• SAMBA server
  • /tftpboot/pxes shares for imaging
• Node image repository
  • himrod.cs:/himrod.node - current image of working nodes on himrod
• Has private NAT firewalled network - 192.168.1/24 10.0.15/24
  • DHCP with local name server
  • PXE tftpboot functions
    • Boot Linux repair and imaging utilities or Linux network installers
  • DNS name server for NAT
  • SAMBA file shares
  • NFS file shares
  • APACHE Web service
    • DNSMASQ lease status

Himrod Setup and Configuration overview files

• Software under /cscf-adm/src provides a number of tools for setting up himrod
  • NETWORKING
    • /etc/network/interfaces network interfaces for system
      • These MUST be statically defined for our setup scripts to work
    • /cscf-adm/src/hosts/common_hosts.himrod*
      • The master Makefile in /cscf-adm/src copies this to /usr/local/bin/common_host
      • EXTIF="em4.529" defines the device name for the extrenal interface
      • INTIF="bond0" main internal network
      • INTNAT="TRUE" use NAT to access the outside
      • INT2IF="em3" secondary internal network
      • INT2NAT="TRUE" use NAT to access the outside
      • Other settings include SAMBA and university network ranges
  • TFTPBOOT PXE
    • /tftpboot/pxes is the PXE root folder
    • /cscf-adm/src/dnsmasq contains all configuration files used for
      • DHCP and DNS
      • /cscf-adm/src/dnsmasq/dnsmasq.common.himrod
        • Defines all himrod nodes and interfaces with simplified IP and MAC address notation
        • provides DNS and reverse DNS for all local networks
        • Example:
          • host himrod
            • himrod.localnet has address 192.168.1.1
          • host himrod.cs
            • himrod.cs has address 129.97.152.5
      • make processes /cscf-adm/src/dnsmasq/dnsmasq.common.himrod
        • Creates:
          • /etc/hosts - defines local addresses for all himrod nodes
          • /etc/resolv.conf using /cscf-adm/src/dnsmasq/resolv.template
          • /etc/dnsmasq.hosts - is an include file in /etc/dnsmasq.conf
            • Note: /etc/dnsmasq.conf ONLY includes dnsmasq.hosts - do not define anything else
      • TFTPBOOT

• • FIREWALL and NAT for internal private networks - optionally provides NAT
  • SAMBA shares - also used by imaging tools
  • NFS for live boot images

Main configuration and setup directories

Networking

• /etc/network/interfaces - system network configuration
  • The setup scripts assume the network setting exist statically in order to work

Himrod Setup and Installation folders

• /cscf-adm/src
  • This contains all of the tools used to do the initial setup and configuration of himrod
• /cscf-adm/src - program sources
• /cscf-adm/src/dnsmasq - DNSMASQ / TFTPBOOT /DNS serices and configuration
• /cscf-adm/src/pxe - PXE boot files - a minimal working PXE tree with boot images
• /cscf-adm/src/hosts - host network configuration - defines interfaces used by all scripts*
• /cscf-adm/src/syslinux - Syslinux sources - used for PXE booting
• /cscf-adm/src/Idrac - iDrac scripts - Dell configuration and licencing scripts
• /cscf-adm/src/cluster - Cluster scripts
• /cscf-adm/src/twiki - Twiki documents - autogenerated
• /cscf-adm/src/html - HTML documents - autogenerated
• /cscf-adm/src.web - web based reporting utils

PXE / TFTPBOOT files and descriptions

/tftpboot/pxes

• /tftpboot/pxes - PXE root directory Add any boot images under this directory - see below for specific directories that are already defined

/tftpboot/pxes/pxelinux.0

• PXE bootstrap file from syslinux - all systems boot this file first

/tftpboot/pxes/pxelinux.cfg/

• Top level PXE menu files go here - defined in binary file pxelinux.0

/tftpboot/pxelinux.cfg/default

• Top level PXE boot menu

/tftpboot/pxelinux.cfg/default.conf

• default menu include used by al menus
  • defines vesa display mode and resolution, defaulyt timeouts and prompts
  • you should not ever have to change this

/tftpboot/pxelinux.cfg/UW.png

• default graphic background for menus

/tftpboot/pxelinux.cfg/clonezilla_menu

• Clonezilla PXE boot menu
  • Primary imaging tool

/tftpboot/pxelinux.cfg/acronis_menu

• Acronis imaging tools menu

/tftpboot/pxelinux.cfg/rip_menu

• Recover is Possible RIP boot images
• there are rip_menuNN.N menus for each version
  • We used this tool to image the cluster - to be replaced by clonezilla

/tftpboot/pxelinux.cfg/dos_menu

• not used on himrod

/tftpboot/pxelinux.cfg/diag_menu

• not used on himrod

/tftpboot/pxelinux.cfg/live_menu

• Live boot images of Ubuntu

/tftpboot/pxes/syslinux

• Contains syslinux ustilities used by the menu system

/tftpboot/pxes/utils/

• used by dos_menu and diag_menu - not used on himrod See pxelinux.cfg/default

/tftpboot/pxes/RIP/

• Recovery is Possible - locally updated version of RIP PXE images

/tftpboot/pxes/avast

• Avast antivirus components - added to RIP images

/tftpboot/pxes/bin

• bin files added to RIP images

/tftpboot/pxes/clonezilla

• Colonezilla PXE boot images
• bin files added to RIP images

/tftpboot/pxes/images

• General purpose images folder free to use for deplyment images

/tftpboot/pxes/iso

• Ubuntu live iso shares used for PXE/NFS live booting

/tftpboot/pxes/pxelinux.cfg/ghost.src

• Linux Ghost code with additions to work with RIP.

/tftpboot/pxes/ununtu-installers

• Ubuntu network installers

Key files and roles

• /cscf-adm/src/hosts/common_hosts.himrod defines interfaces and roles (ie is the interface to a NAT network)

IDrac Scripts

--+++ Reset node using iDrac

• /cscf-adm/src/IiDrac/reset_node ilom-NODE - where NODE is lised in the table above

Apply iDrac License Keys to machines

• ./allnodes_apply_idrac_keys
  • Uses service_tags.txt for list of nodes - and to match the license file
    • Assumes license files are in current directory - see extract_keys

Setup boot drive on nodes himrod-1 .. 23

• ./allnodes_setup_boot_hd
  • Calls setup_boot_hd with node name

Setup disks on nodes himrod-1 .. 23

• ./allnodes_setup_disks
  • Calls setup_disks with node name

Setup node disks via iDRac

• ./setup_boot_hd nodename [-r]
  • nodename is like ilom-himrod-3
  • -r implies reset node

Update all nodes firmware

• ./allnodes_update_firmware
  • Uses service_tags.txt for list of nodes
  • Calls update_firmware

Update node firmware

• ./update_firmware nodename R620|R720xd
  • nodename is like ilom-himrod-3

Extract Dell license keys

• ./extract_keys
  • Extracts the License.zip file , then renames the license files so that the service tag for each node is part of the file name.
    • Example: DE00000040318201-1DF2DZ1.xml

List iDrac firmware versions for all nodes

• ./get_idrac_versions
  • Uses service_tags.txt for list of nodes

List node disks

• ./list_disks nodename
  • Lists iDrac disk and controller names on a node
  • nodename is like ilom-himrod-3

Reboot node into PXE - onceupdate_rip

• ./reboot_once_pxe nodename
  • nodename is like ilom-himrod-3

Reset node

• ./reset_node nodename
  • nodename is like ilom-himrod-3

Reset nodes

• ./reset-nodes
  • Resets all nodes except himrod-1 and storage nodes*

iDrac Help on the web

• Dell Remote Access Controller - DRAC / iDRAC
  • http://en.community.dell.com/techcenter/systems-management/w/wiki/3204.dell-remote-access-controller-drac-idrac.aspx
• RACADM Command Line Interface for DRAC
  • http://en.community.dell.com/techcenter/systems-management/w/wiki/3205.racadm-command-line-interface-for-drac.aspx

iDrac Directories

/cscf-adm/src/iDrac

• iDrac-Firmware-R620
  • Firmware
• iDrac-Firmware-R720xd
  • Firmware
• R620
  • Documents for R620
• orig
  • Backup of license files in zip format
• docs
  • iDrac documents integrated-dell-remote-access-cntrllr-7-v1.30.30_User's Guide_en-us.pdf

IDrac files

• account iDrac access password
• license_portal_url.txt
  • URL and userid for Dell license portal (This is where we downloaded our iDrac keys)
• racadm_readme.txt
  • Summary of iDrac commands (most of them listed below)
• service_tags_license_names.txt
  • License file n Resets all nodes except himrod-1 and storage nodes*

ames with node names and hardware type R620 or R720xd

• service_tags.txt
  • node name, service tag, hardware type R620 or R720xd
• Service Tags ver 4-1-2-20140605.xlsx
  • file with node names and service tags

iDRac files on tftpboot server

• /tftpboot/pxes/firmware/R620IDRAC/firmimg.d7
  • Firmare for R620 and R720xd - the happen to be the SAME image

IDrac ILOM network configurration examples

• • Set NIC to ILOM4
    • racadm config -g cfgLAnNetworking -o cfgNicSelection 5
  • Get NIC
    • racadm getconfig -g cfgLanNetworking -o cfgNicSelection
  • Static config
    • racadm setniccfg -s 172.19.128.254 255.255.255.0 172.19.128.1
  • Get NIC info - will list the ILOM and which interface it ties to
    • racadm getniccfg
  • Enable IPMI over lan
    • racadm config -g cfgIpmiLan -o cfgIpmiLanEnable 1
  • DHCP
    • racadm setniccfg -d

Idrac Firmware commands

* See update_firmware script - we flash update from the tftpboot server

• -p : upload firmware image from the client (RACADM)
• -g : use TFTP download process for firmware update (use with -u)
• -u : perform firmware update operation (used with -p and -g)
• -a : specifies TFTP server IP address used for firmware image (used with -g)
• -d : specifies source path where firmware image exists (default is local)
• -f : use FTP download process for firmware update
• -s : query for current firmware update process status
• -r : rollback to the standby firmware
• --clearcfg : after firmware update, removes the previous idrac configuration

Firmware update examples

* See update_firmware script - we flash update from the tftpboot server

• Upload a firmware image from the client and start firmware update:
  • racadm fwupdate -p -u -d /tmp/images
• Upload firmware image from FTP server and start firmware update:
  • racadm fwupdate -f 192.168.0.10 test test -d firmimg.d7
• Upload firmware image from TFTP server and start firmware update:
  • racadm fwupdate -g -u -a 192.168.0.100 -d /tmp/images
• Query the current status of the firmware update process:
  • racadm fwupdate -s
• Rollback to the standby firmware:
  • racadm fwupdate -r
• Upload firmware image from TFTP server, start firmware update. After firmware update is complete, delete previous iDRAC configuration:
  • racadm fwupdate -g -u -a 192.168.0.100 -d /tmp/images --clearcfg

ipmitool networking

• ipmitool
• list users - note each user has an number id
  • ipmitool user list 1
• set user password
  • ipmitool user set 2
• test user password - not you might want to be carefull - this password is on the command line!
  • ipmitool user test 2 16
• display ilom network settings
  • ipmitool lan print 1
• set ilom dhcp
  • ipmitool lan set 1 ipsrc dhcp
• set static address
  • ipmitool lan set 1 ipsrc static
  • ipmitool lan set 1 ipaddr 172.19.128.254
  • ipmitool lan set 1 defgw ipaddr 172.19.128.
• Get current active ilom network port
  • ipmitool -I open delloem lan get active
• Set ilom network port ( on iDRAC12g OR iDRAC13g : dedicated, shared with lom1, shared with lom2,shared with lom3,shared with lom4,shared with failover lom1,shared with failover lom2,shared with failover lom3,shared with failover lom4,shared with Failover all loms, shared with Failover None))
  • ipmitool -I open delloem lan set dedicated
• Get current ilom network port
  • ipmitool -I open delloem lan get

--++ Imaging a node

Live boot images

• /cscf-adm/src/fetch_ubuntu_releases 14.04 will update all of the latest 14.04 releases
  • Any number of version numbers can be listed - they will all be updated
  • calls pxe_liveboot_nfs service to remount and export new images

PXE Ubuntu Installers

• /cscf-adm/src/fetch_ubuntu_installers - creates a PXE installer for all liveboot images
• This is only of use for an install from scratch

RIP PXE node imaging

• We used RIP PXE boot to image the original nodes on himrod - the suggested new was is to use clonezilla
• Use iDRac web console for node
  • firefox --no-remote ilom-himrod-1
  • login with root and 2013 cscf-adm passwd
• Network boot the node in pxe mode: reboot_once_pxe
• Choose RIPLinuX 13.7
  • RIPLinuX 13.7 64Bit with Kernel from himrod version vmlinuz...
• login root - no password
• setup network dhcpcd
• run clone script: distroy_and_clone
  • This reformats /dev/sda partitions and copies /himrod-node (from himrod) to / on the node
• cd /mnt/sda1/ubuntu-scripts
• chroot and bind required filesystems into /mnt/sda1:
  • ./mnntpart /dev/sda1 - ignore any bind mount errors - a few are for other versions of ubuntu
• verify networking, ie: ping www.yahoo.com
• fix grub
  • cd /ubuntu-scripts
  • ./grubfix
    • yes remove grub - we purge and reinstall
    • boot */dev/sda1

RIP updates or customizati"> Local RIP updates or customization

• /cscf-adm/src/update_rip 13.7 pxe

USAGE: ./update_rip version  type
   version is numeric
   type is grub,grub2,syslinux,pxe or all
Example: ./update_rip 13.7 grub2
   patches RIP13.7 directory with ghost
Example: ./update_rip all
   updates ALL (pxe,syslinux,grub2) boot images on himrod

• This script will also add cscf specific customizations to RIP - it also rewrites the /tftpboot/pxes/pxelinux.cfg/rip_menu and rip_menu13.7 etc

Clonezilla PXE boot

• /cscf-adm/src/fetch_clonezilla updates to the latest version automatically
  • /tftpboot/pxes/pxelinux.cfg/clonezilla is the PXE boot menu - does not have to change with most updates.
  • We have the latest version (as of 10 April 2015) of clonezilla on himrod
• Web site: http://clonezilla.org/
• Screen shots
  • http://clonezilla.org/screenshots/?in_path=/00_Clonezilla
• Features
  • Can image and restore both Windows and Linux systems
  • Recover/backup
  • PXE bootable
  • Multicast
  • Supports a wide range of file systems
  • Backup only parts of the disk that are used
• Limitations
  • Cannot restore to smaller disks or partitions