CSCF Application Server (Asimov) notes

19 Aug 2015 - Updates to new server notes are in progress - mostly done

Summary

Purpose

• File system for CSCF image deployment and misc services
• Prototype for next generation Teir 2 File Server

Hardware

• SuperMicro CSE-847E16-R1K28LPB case
• SuperMicro X9DRD-7LN4F-JBOD-O motherboard, 2xIntel Xeon E5-2630 V2 (6 Core) 2.6Ghz, 128GB 1600Mhz DDR3 ECC RAM (16x8GB)
• 2xIntel 100GB SSD (OS RAID 1)
• 24x Seagate Constellation 4TB SAS (ST4000NM0023)
• 800G PXIE flash disk - zcache for zfs

Ubuntu OS

• We are now running Ubuntu 14.04.2 LTS on new hardware

Backups

• The system is backed up by Legato Networker
• NOTE: files under /backup - ARE NOT backed up! - this location is used for TEMPORARY storage only
  • Typical use would be as a temporary place to put files while reinstalling a CS machine.

Data migration - old to new Asimov

• AsimovMigration - data migration document

Index

Feature Summary - high level

• Ubuntu Linux OS 14.04.2
• Joined to CS AD - cscf-adm and cscf-op are local users, Samba passwords are local
• Software and CD installation repository shared on the network using SAMBA
  • 285 CD's, Windows and Linux, that can be installed over the network
    • CD's are in RAW ISO format - easy to burn
    • CD's ram images are also mounted online so they are all available
  • Common Windows Applications and License Keys
• SAMBA server - see CSApplicationServer#File_Shares
• NFS server on 192.168.1.0/24
• System Imager - see SystemImager
• Machine image repository - Clonezilla, Acronis and GHOST images of fully installed machines
  • See ImageDeploy
  • ie "Syspreped" Windows machine with Applications fully configured/patched and/or fully setup Linux images
    • Most images are created and maintained by the CORE infrastructure group - Phil Beldowski, etc
• Has private NAT firewalled network - VLAN 420
  • DHCP with local name server - See PXE server
  • PXE tftpboot functions - See PXE server
    • Boot DOS and Linux repair and imaging utilities or Linux network installers
  • DNS name server for NAT - See PXE server
  • SAMBA file shares
  • SSH and SSHFS - we can ssh mount resources
  • NFS file shares
  • APACHE Web service
    • TODO Many of the SAMBA shares can be accessed here as http://asimov:8080/exports or /iso
    • DHCP reporting tool for private network activity
• External network
  • SAMBA file shares - to all of UofW
  • SSH and SSHFS - we can ssh mount resources
  • NFS file shares - limited to internal networks
  • APACHE Web service
    • TODO Many of the SAMBA shares can be accessed here as http://asimov:8080/exports or /iso

Image deployment over the network from ASIMOV

• Networked PowerQuest and GHOST image install using ASIMOV ImageDeployment
• Alternate NETBOOT floppy/CD boot for network access to ASIMOV WindowsNetBoot
• See: ImageNamingRules for notes on images are named and what they mean.
• See: CscfGradImageNotes for image-specific creation and post-install notes

Drive and File System

• asimov-pool/images 57T 1.2T 56T 2% /images
• asimov-pool/coregroup 60T 3.7T 56T 7% /coregroup
• asimov-pool/backups 57T 1.3T 56T 3% /backups

SMART Disk Monitoring

• TBD

Modifying service run levels

• update-rc.d

APC UPS

Note: currently disabled

• Config: /etc/apcupsd/apcupsd.conf

File Shares

• To see a list of shares you can run
  • NET VIEW command from a command prompt in Windows, for example, NET VIEW \\ASIMOV
  • smbclient -L asimov.cscf -U cscf-adm from Linux (provided by the smbclient package in Ubuntu).

root@asimov-new:~# smbclient -L localhost -U cscf-adm
Enter cscf-adm's password: 
Domain=[CS_GENERAL] OS=[Unix] Server=[Samba 4.1.6-Ubuntu]

   Sharename       Type      Comment
   ---------       ----      -------
   homes           Disk      Home Directories
   print$          Disk      Printer Drivers
   PXE             Disk      PXE Boot home directory
   PXEIMAGES       Disk      PXE Boot share for Network Book Images
   PXEISO          Disk      Mounted CDROM Images,some may be Licensed Software, Requires License form IST
   PXERAWISO       Disk      RAW ISO CDROM Images, some may be Licensed Software, Requires License form IST
   OPT             Disk      OPT utilities
   research        Disk      Research Share
   install         Disk      Lawrence Folland - Research Group - work on file reorg.
   CORE            Disk      CORE GROUP
   CORE_OLD        Disk      CORE GROUP OLD
   IMAGES          Disk      Deployment images for all of CSCF
   PROJECTS        Disk      CSCF Projects
   iso             Disk      Mounted CDROM Images,Licensed Software, Requires License form IST
   rawiso          Disk      RAW ISO CDROM Images, Licensed Software, Requires License form IST
   exports         Disk      ALL EXPORTS, Microsoft OS Dependent, Some site license software
   software        Disk      SOFTWARE not provided on CDROM images, Some licensed software
   sysprep         Disk      SYSPREP INSTALL IMAGES - Should move to IMAGES share
   SUPERMICRO      Disk      SUPERMICRO Utilities IPMI, Fimware,etc
   OFFICE10        Disk      Alias Office XP, network install, Note: needs license forms
   OFFICE11        Disk      Alias Office 2003, network install, Note: needs license forms
   vs.net2003      Disk      Alias VISUAL STUDIO NET 2003, Note: needs license forms
   vs.net2005      Disk      Alias VISUAL STUDIO NET 2005, Note: needs license forms
   backup          Disk      CSCF - RESEARCH USE - GENERAL TEMP BACKUP FOR ANY COMPUTER
   MAC             Disk      MAC software
   CLAYTON         Disk      Clayton Server Images and scratch area
   ADBACKUPS       Disk      AD Backups
   nexus           Disk      STEVE NICKERSON - NEXUS
   nexus_adm       Disk      STEVE NICKERSON - NEXUS ADMIN
   SNICKERS        Disk      Steve Nickerson image storage area
   IPC$            IPC       IPC Service (asimov-new server (Samba, Ubuntu))
   cscflp01        Printer   cscflp01
   xrxpr003        Printer   xrxpr003
   xrxpr004-monochrome Printer   xrxpr004-monochrome
   ljc-alg         Printer   ljc-alg
   dbpr001         Printer   dbpr001
   dbpr004         Printer   dbpr004
   xrxpr005        Printer   xrxpr005
   xrxpr004-colour Printer   xrxpr004-colour
   xrxpr002        Printer   xrxpr002
   cs006437        Printer   cs006437
   ljc-dc3334      Printer   ljc-dc3334
   lj-m425dn-dc2313 Printer   lj-m425dn-dc2313
   lj-dc3624       Printer   lj-dc3624
   lp2-dc3507      Printer   lp2-dc3507
   chilpr01        Printer   chilpr01
   mfp102          Printer   mfp102
   lj-cs-simplex   Printer   lj-cs-simplex
   lj-dc2582       Printer   lj-dc2582
   cs006417        Printer   cs006417
   slush           Printer   slush
   ljp-maple       Printer   ljp-maple
   xrxpr001        Printer   xrxpr001
   lp7-cs          Printer   lp7-cs
   cs007854        Printer   cs007854
   docutech6180    Printer   docutech6180
   graphics-dc-xerox-5775 Printer   graphics-dc-xerox-5775
   lj-dc3334       Printer   lj-dc3334
   cs007582        Printer   cs007582
   graphics-gsc-ricoh Printer   graphics-gsc-ricoh
   lp-cscf-private Printer   lp-cscf-private
   lj-dc2306       Printer   lj-dc2306
   xrxpr006        Printer   xrxpr006
   graphics-dc-ricoh Printer   graphics-dc-ricoh
   ljp-plg         Printer   ljp-plg
   lj-cs           Printer   lj-cs
   lj1-isg         Printer   lj1-isg
   lj2-watform     Printer   lj2-watform
   lp-5136a        Printer   lp-5136a
   cscf-adm        Disk      Home Directories
Domain=[CS_GENERAL] OS=[Unix] Server=[Samba 4.1.6-Ubuntu]

   Server               Comment
   ---------            -------
   ASIMOV-NEW           asimov-new server (Samba, Ubuntu)

   Workgroup            Master
   ---------            -------
   CS_GENERAL           

Common Sources

• Source directory cscf-adm/src/src

Source files are located in three directories

• /cscf-adm/src/dnsmasq - DHCP,PXE,DNS configuration files and scripts - some host specific
  • See DNSMASQ PXEBootFilesExample NatMasqFirewallExample
  • See PXE server
• /cscf-adm/src - program sources
• /cscf-adm/src/host - some host specific configuration
• /cscf-adm/scripts - backup and other scripts
• /cscf-adm/src.web - web based reporting utils

Common System Services

Notes System services can be controlled with the service command which is a script that simply invokes command in /etc/init.d . Each script has standard options such as start, stop, restart, status

• mount_cds - mounts /iso images created from /images/rawiso
• samba - samba service
• apache2 - Web Server
• dnsmasq - - Local Name server,DHCP server,PXE server * See DNSMASQ for details * See PXE server
• firewall NatMasqFirewallExample
• tftpboot structure See PXEBootFilesExample
• Examples
  • service mount_cds stop - removes /iso
  • service mount_cds start - creates /iso loopback mounts from /images/rawiso

Setup Detail

• Software under /cscf-adm/src provides a number of tools for setting up asimov
  • NETWORKING
    • /etc/network/interfaces network interfaces for system
      • These MUST be statically defined for our setup scripts to work
    • /cscf-adm/src/hosts/common_hosts.asimov*
      • The master Makefile in /cscf-adm/src copies this to /usr/local/bin/common_host
      • EXTIF="em1" defines the device name for the extrenal interface
      • INTIF="eth5" main internal network
      • INTNAT="TRUE" use NAT to access the outside
      • INTIF2="p6p1" secondary internal network
      • INTNAT2="TRUE" use NAT to access the outside
      • INTIF3="em3" third internal network
      • INTNAT3="TRUE" use NAT to access the outside
      • Other settings include SAMBA and university network ranges
  • TFTPBOOT PXE
    • /tftpboot/pxes is the PXE root folder
    • /cscf-adm/src/dnsmasq contains all configuration files used for
      • DHCP and DNS
      • /cscf-adm/src/dnsmasq/dnsmasq.common.asimov
        • Defines all asimov nodes and interfaces with simplified IP and MAC address notation
        • provides DNS and reverse DNS for all local networks
        • Example:
          • host asimov
            • asimov.localnet has address 192.168.1.1
          • host asimov.cs
            • asimov.cs has address 129.97.15.251
      • make processes /cscf-adm/src/dnsmasq/dnsmasq.common.asimov
        • Creates:
          • /etc/hosts - defines local addresses for all asimov nodes
          • /etc/resolv.conf using /cscf-adm/src/dnsmasq/resolv.template
          • /etc/dnsmasq.hosts - is an include file in /etc/dnsmasq.conf
            • Note: /etc/dnsmasq.conf ONLY includes dnsmasq.hosts - do not define anything else
      • TFTPBOOT

• • FIREWALL and NAT for internal private networks - optionally provides NAT
  • SAMBA shares - also used by imaging tools
  • NFS for live boot images

Initial Setup overview

Only Done once

• Edit /cscf-adm/src/dnsmasq/dnsmasq.common.asimov
  • This defines subnets and host/mac IP assignments
• Edit /etc/network/interfaced
  • define ALL networks statically
• Edit /cscf-adm/src/hosts/common_hosts.asimov
  • define interface names and roles
• Run /cscf-adm/src/install_packages
  • This installes required packages
• Run make all
  • runs install scripts
    • dnsmasq, samba, nfs setup sccf-adm accounts
  • runs update scripts
    • download pxe live images etc
  • installs NAT firewall

Setup Updates

TODO move asimov tools example into generic example

• Run: make copy_scripts
  • installs common_functions, common_host and common_vars in:
    • /usr/local/bin
• Run make docs
  • Creates the TWIKI and HTML documenation from sources places them in:
    • /cscf-adm/src/twiki and /cscf-adm/src/html

Main configuration and setup directories

Networking

• Configuration Files
• /cscf-adm/src/hosts/common_hosts.asimov defines interfaces and roles (ie is the interface to a NAT network)
  • /etc/network/interfaces - system network configuration
  • The setup scripts assume the network setting exist statically in order to work
  • /cscf-adm/src/dnsmasq/Makefile
  • /cscf-adm/src/dnsmasq/dnsmasq.common.asimov
  • /cscf-adm/src/hosts/common_hosts.asimov defines interfaces and roles (ie is the interface to a NAT network)
  • /etc/hostname
  • /etc/hosts
  • /etc/resolv.conf

Asimov Setup and Installation folders

• /cscf-adm/src
  • This contains all of the tools used to do the initial setup and configuration of asimov
• /cscf-adm/src - program sources
• /cscf-adm/src/dnsmasq - DNSMASQ / TFTPBOOT /DNS serices and configuration
• /cscf-adm/src/pxe - PXE boot files - a minimal working PXE tree with boot images
• /cscf-adm/src/hosts - host network configuration - defines interfaces used by all scripts*
• /cscf-adm/src/syslinux - Syslinux sources - used for PXE booting
• /cscf-adm/src/Idrac - iDrac scripts - Dell configuration and licencing scripts
• /cscf-adm/src/cluster - Cluster scripts
• /cscf-adm/src/twiki - Twiki documents - autogenerated
• /cscf-adm/src/html - HTML documents - autogenerated
• /cscf-adm/src.web - web based reporting utils

DNSMASQ /cscf-adm/src/dnsmasq

• Configuration Files
  • /cscf-adm/src/dnsmasq - see DNSMASQ
• dnsmasq is a single package that provides PXE BOOT, DNS, DHCP services

Makefile

• make is the only file run by and admin to update asimovs dnsmasq settings
• compiles mkhosts.c
• runs install_dnsmasq
• Note: make can be run at ANY time to make sure all files are up to date

install_dnsmasq

• Only called from make ( ie. Makefile ) to install and configure dnsmasq using files defined in the following sections
• Stops nscd and dnsmasq
• updates /etc/resolv.conf using resolv.template files
• updates /etc/dnsmasq.hosts usings hsosts.template
• creates /etc/hosts , /etc/resolv.conf and /etc/dnsmasq.hosts
  • Updates /etc/dnsmasq.conf to include /etc/dnsmasq.hosts
• creates /etc/Networkmanager/dispatcher.d/00-dnsmaq trigger script to rewrite /etc/resolve.conf in case Network Manager is installed
  • This prevents other programs from messing up /etc/resolv.conf
• starts nscd and dnsmasq

DNSMASQ Config files

dnsmasq.common.asimov

• asimov specific configuration for dnsmasq
• Put all IP,MAC address and interface information for the cluster here

dnsmasq.common

• just a symlink to dnsmasq.common.asimov created by make install process

DNSMASQ Template Files

• * These files are preprocessed and added to the top of the automatically generated /etc/dnsmasq.hosts file
  • dnsmasq/dnsmasq.template.private
    • default DNSMASQ settings for ONE private network
  • dnsmasq/dnsmasq.template.private_two
    • default DNSMASQ settings for TWO private networks
  • dnsmasq/dnsmasq.template.private_three
    • default DNSMASQ settings for THREE private networks
  • dnsmasq/hosts.template
    • defines fixed records we always want in the /etc/hosts file
  • dnsmasq/resolv.template
    • defines fixed record we want added to the top of the /etc/resolv.conf file
    • Default external DNS servers - should be moved into common_host

DNSMASQ Compiled files

• To recreate these files:
  • cd /cscf-adm/src/dnsmasq
  • make

/etc/hosts

• Create from mkhosts and install_dnsmasq using dnsmasq.common.asimov and hosts.template

/etc/resolv.conf

• Create from mkhosts and install_dnsmasq using resolv.template

/etc/dnsmasq.hosts

• Create from mkhosts and install_dnsmasq using dnsmasq template files

dnsmasq.compiled.asimov

• The top level include for dnsmasq created by the make install process
• includes preprocessed dnsmasq.template files
• includes dnsmasq.common

DNSMASQ mkhosts and mkhosts.c

• preprocessor program that reads
  • dnsmasq.compiled.asimov and hosts.template
  • Writes /etc/dnsmasq.hosts - a file included from the otherwise totally blank /etc/dnsmasq.conf file
  • Write /etc/hosts
  • Creates DHCP entries, DNS records - both forward and reverse lookups for local networks

mkhosts Ducumenation

• What: mkhosts.c a preprocessor for dnsmasq configuration files
• The preprocessor handles creating reverse dns records and settings from
• a simpilfied syntax
• Syntax:
• ttl= time
• time is a ttl as per the dnsmasq man pages
• Example: ttl=1440m - set the default ttl to 1440 minutes until the next ttl keyword
• * subnet= subnet specify the subnet
• subnet is a subnet specification - LIMITATIONS mkhosts uses hard coded /24 network - first 24 bits
• Example: subnet=192.168.1 - this defines the subnet the same as 192.168.1.0 .. 192.168.1.255
• ip,*mac*,*cname* [,*cname*]...
• ip is the last digit of the subnet only
• mac is the MAC address of the interface you want to assign the address to
• cname a common separated list of cnames
• Example: 101,00:25:90:29:eb:ec,m160-1-local-comm,m160-1
• * Creates

•    *     dhcp-host=192.168.2.101,00:25:90:29:eb:ec,m160-1-local-comm,1440m
     *     ptr-record=101.2.168.192.in-addr.arpa.,"m160-1-local-comm"
     *     address=/m160-1-local-comm/192.168.2.101
     *     cname=m160-1,m160-1-local-comm
     *     

• prefix= We can redefine dhcp name prefix for reverse lookups
• iface= just puts a comment in the output file
• tag= defind tags to be appened to dhcp-host
• pxelinux= Define the pxelinux subnet - hosts under this section can get a private boot file
• Each mac address will simlink to the file pxelinux.cfg/nodes
• include= process an include file
• Example:
• ttl=1440m
• subnet=192.168.2
• 1,00:25:90:0e:5b:a2,m160-local-comm,m160-comm,m160
• 101,00:25:90:29:eb:ec,m160-1-local-comm,m160-1
• ...

BASH Scripts summary

• accounts - default cscf-adm and cscf-op encrypted passwords
• cleanup - Delete files under a directory DIR and children that are older the DAYS old
  • ./cleanup directory days
• common_functions
  • Common shell scripts used by most scripts under /cscf-adm/src
• common_vars
  • Common directory locations and definitions
• common_host
  • compiled into /usr/local/bin from /cscf-adm/src/hosts/common_hosts.asimov
• cscf-root-pub-key.txt
  • copy of CSCF root public key to be added to /root/.ssh/autorized_keys2 file
• distroy_and_clone
  • asimov specific scripit added to RIPimage
  • Partitions and formats all drives on a node - mounts /dev/sda1 on /mnt/sda1
  • Runs: rsync -a -x -H --numeric-ids homrod:/asimov.node/ /*

. * dnsmasq.sh - lists dnsmasq leases

• fetch_clonezilla - grabs lates clonezilla image and installes it in /tftpboot/pxes/clonezilla
  • see: /tftpboot/pxes/pxelinux.cfg/clonezilla_menu
• fetch_pxe
  • Capture existing PXE code and images for the Image Deployment System
  • Not used on asimov
• fetch_rip
  • Downloads lates version of RIP
• fetch_ubuntu_installers
  • Downloads Ubuntu PXE instalers for any iso images under /tftpboot/pxes/rawiso/Ubuntu
  • Updates /tftpboot/pxes/ubuntu-installer/menu.conf called from master pxelinux menu
• fetch_ubuntu_releases
  • Downloads ubuntu iso images to /tftpboot/pxes/rawiso/Ubuntu
  • Example: ./fetch_ubuntu_releases 12.04 14.04 beta*
• firewall * NAT firewall service for system
  • Uses /usr/local/bin/common_host and helper scripts common_vars and common_functions
• fixhost
  • change hostname and ssh keys for a host - update postfix
• fixperms
  • Update the permissions on files under a directory
  • Only update those files that have them wrong so that backups do not update files needlessly
• flush_cache
  • This script will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.
• format_disks
  • Partition and Format disks on a asimov node to prepare for imaging
• grub_fix
  • Updates /etc/default/grub, purges then reinstalls grub
• install_admin_users
  • Add local Administrative and Samba users defined in $ADM_USERS - see common_host
• install_dell_utils
  • Install Dell OpenManage Ubuntu Repository - not used on asimov
• install-networker
  • Install Legato Networker for CSCF backup services
• install_packages
  • Install common Ubuntu packages required for the Image Deployment System
• install_pxe
  • Install the PXE code and images for the Image Deployment System
  • We only install install the files once from the pxe folder
  • We should consider merging updates FROM the pxe folder to the pxe folder or the reverse
  • Correctly we use fetch_pxe (this excludes files that can be regenerated)
  • make fetch_pxe will do this
• install_samba
  • setup cscf-adm and cscf-op local user
    • Prompts to create an iinitial samba password for each account
• install_syslinux
  • Install and compile syslinux (the default Ubuntu version is out of date)
  • Installs files in /tftpboot/pxes/syslinux
• install_ubuntu_scripts
  • not used on asimov
• IPMIview
  • Supermicro IPMI utility wrapper
  • Console access and IPMI settings
• lc.c and lc
  • UofW list catalog program
• list_samba_users
  • List current samba users
• Makefile
  • Master make file calls all install and update scripts
• mntpart
  • used to chroot mount a partition - used by recovery tools like RIP
• mntprox
  • bind mounts partitions into chrooted environment
• NODES
  • Defines all of the cluster nodes by name and ILOM name
• pxe2usb.sh
  • Converts asimov PXE boot enviornment into standalone USB KEY
• pxe_liveboot_nfs
  • Service to mount Live Boot images
• readiso
  • Read an CD or DVD and create an ISO image
• sh2twiki and sh2twiki.c
  • Creates TWIKI and HTML documenation form simplified Markup Language
• syslinux-6.02.tar.gz
  • Syslinux source
• testfree and testfree.c
  • Used to verfify a partition is mounted and has enough free space
• test_vars
  • Lists variables used by common_vars, common_host and common_functions
• update_fstab
  • Updates /etc/fstab on nodes to include NFS home directories
• update_rip
  • Apply local costomizations to local RIP images
• updates
  • Add additional repositories, NTP and SMTP configuration, created symlinks need for CS AD
  • Used on clean system after first install
• wakeup uses wake on lan feature to wakeup any host listed in /etc/dnsmaqsq.hosts

firewall script provides firewall and NAT services for a private network

• Notes:
  • If you need to add or limit existing service rules
  • You should only have to make changed to the functions:
  • local, and less likely; mail_firewall

check()

• What:
  • Must have a NAT defined or die
• Return:
  • exit 1 if no firewall is defined

init_firewall()

• What:
  • remove all firewall rules and kernel modules
  • sets state to ACCEPT or DROP
• Arguments:
  • allow - or - drop
  • allow = ACCEPT all traffic
  • drop = DROP all traffic

list()

• What:
  • List network interfaces

hosts_allow()

• What:
  • Modifies /etc/hosts.allow adding the private networks

local_rules()

• What:
  • Local default firewall rules for UW
• Notes:
  • Customize these to suit your needs

main_firewall()

• What:
  • Main firewall code - site independent firewall code
• Notes:
  • Calls local_rules

add_ssh_host()

• What:
  • Open an inbound ssh rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

del_ssh_host()

• What:
  • Close an inbound ssh rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

checkhost()

• What:
  • Check if a host has a firewall rule
• Arguments:
  • host
• Returns:
  • Display iptables result

openhost()

• What:
  • Open an inbound rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

blockhost()

• What:
  • CLose an inbound rule for the specified external host
• Arguments:
  • host
• Returns:
  • Fatal if host not specified

install_service()

• What:
  • Install firewall script as a service
• Notes:
  • Sets the firewall state to allow all
• Returns:
  • Displays status

remove_service()

• What:
  • Remove firewall service
• Notes:
  • Sets the firewall state to allow all
• Returns:
  • Displays status

Main Code Section:

• Arguments:
  • start - start firewall
  • stop - stop firewall
  • restart - restart firewall
  • status - firewall status display
  • add_ssh_host - add inbound allow ssh rule for host
  • del_ssh_host - del inbound allow ssh rule for host
  • openhost - add inbound allow open rule for host
  • blockhost - add inbound block rule for host
  • checkhost - check host rule
  • install - install this script as a service
  • uninstall

C Programs

• basestr.c
  • :Match substring
  • Returns: return( strncmp(argv[1],argv[2],len) == 0 ? 0 : 1);
• fixname.c
  • Escape strings so they can be safely passed to a shell
  • Names may be quoted with ''s
• lc.c
  • University of Waterloo List Catalogue command
• sh2twiki.c
  • Convert BASH and C markup to HTML or TWIKI
• testfree.c
  • Usage: testfree device -v -f threshold(in Megabytes)

Automatic Documentation

• make docs will automatically extract BASH and C markups
  • Creates TWIKI or HTML Documentation from Markup
• Features
  • Uses half the markup required the doxygen
  • Understands SHELL Variables and Functions

Markup Rules

• Heading: Markup syntax notes
• Comments:
  • Comments start with // or #
  • Examples
  • # this is a comment
  • // this is a comment
• Tokens:
  • Tokens are any word following a comment that ends with ':'
  • Examples
  • # Notes:
  • // Arguments: userid result
  • Notes: the token and any following comments are part of the token
• Reserved:
  • The following tokens have special meaning
  • Title: Creates a title for the document.
  • This keyword and any unbroken chain of comments and unreserved
  • tokens are part of the title and will be listed in the index file.
  • Author: Document Author, also Authour works
  • Heading: Create a heading
• Variables:
  • The following code objects have special meaning
  • VAR=value Variable assignments
  • This variable if following a token will become a comment
  • # ======== This a a section break
  • // ======= This is a section break
• Blank_Lines:
  • Blank Comments lines end a token or grouping
  • Examples:
  • //
  • #
  • C enum are processed
  • enum {
  • value1,
  • value2
  • };
• Functions: are processed along with the contents by counting brace levels
• Note: ANY unbroken preceding group of tokens/comments are part of the
  • function documentation.
  • Functions names in the documentation are listed BEFORE the preceding tokens
• Examples:
  • // This is a custom printf command
  • // Arguments: format string, variable number of arguments
  • // Returns: void
  • void myprintf(char *str, ...)
  • {
  • printf code goes here ...
  • }
• Special_Keywords:
• Processed keywords that create documentation sections
  • pause message
  • pause_yes message
  • pause_no message
  • must be followed by if,then,else,fo - optionly end with a section
  • Notes:
  • This is used in the ASIMOV updates script to create TWIKI documentation for
  • That script. The compatibility is included here

PXE

• PXE BOOT /tftpboot/pxes
  • Config: /tftpboot/pxes/pxelinux.cfg/default
  • ISO boot images: /tftpboot/pxes/iso

/tftpboot/pxes/pxelinux.cfg/default

PXE / TFTPBOOT files and descriptions

/tftpboot/pxes

• /tftpboot/pxes - PXE root directory Add any boot images under this directory - see below for specific directories that are already defined

/tftpboot/pxes/pxelinux.0

• PXE bootstrap file from syslinux - all systems boot this file first

/tftpboot/pxes/pxelinux.cfg/

• Top level PXE menu files go here - defined in binary file pxelinux.0

/tftpboot/pxelinux.cfg/default

• Top level PXE boot menu

/tftpboot/pxelinux.cfg/default.conf

• default menu include used by al menus
  • defines vesa display mode and resolution, defaulyt timeouts and prompts
  • you should not ever have to change this

/tftpboot/pxelinux.cfg/UW.png

• default graphic background for menus

/tftpboot/pxelinux.cfg/clonezilla_menu

• Clonezilla PXE boot menu
  • Primary imaging tool

/tftpboot/pxelinux.cfg/acronis_menu

• Acronis imaging tools menu

/tftpboot/pxelinux.cfg/rip_menu

• Recover is Possible RIP boot images
• there are rip_menuNN.N menus for each version
  • We used this tool to image the cluster - to be replaced by clonezilla

/tftpboot/pxelinux.cfg/dos_menu

• not used on asimov

/tftpboot/pxelinux.cfg/diag_menu

• not used on asimov

/tftpboot/pxelinux.cfg/live_menu

• Live boot images of Ubuntu

/tftpboot/pxes/syslinux

• Contains syslinux ustilities used by the menu system

/tftpboot/pxes/utils/

• used by dos_menu and diag_menu - not used on asimov See pxelinux.cfg/default

/tftpboot/pxes/RIP/

• Recovery is Possible - locally updated version of RIP PXE images

/tftpboot/pxes/avast

• Avast antivirus components - added to RIP images

/tftpboot/pxes/bin

• bin files added to RIP images

/tftpboot/pxes/clonezilla

• Colonezilla PXE boot images
• bin files added to RIP images

/tftpboot/pxes/images

• General purpose images folder free to use for deplyment images

/tftpboot/pxes/iso

• Ubuntu live iso shares used for PXE/NFS live booting

/tftpboot/pxes/pxelinux.cfg/ghost.src

• Linux Ghost code with additions to work with RIP

/tftpboot/pxes/ununtu-installers

• Ubuntu network installers

ipmitool networking

• See: ClusterToolsIPMITOOL ---+++ Live boot images
• /cscf-adm/src/fetch_ubuntu_releases 14.04 will update all of the latest 14.04 releases
  • Any number of version numbers can be listed - they will all be updated
  • calls pxe_liveboot_nfs service to remount and export new images

PXE Ubuntu Installers

• /cscf-adm/src/fetch_ubuntu_installers - creates a PXE installer for all liveboot images
• This is only of use for an install from scratch

Clonezilla

• /cscf-adm/src/fetch_clonezilla updates to the latest version automatically
  • /tftpboot/pxes/pxelinux.cfg/clonezilla is the PXE boot menu - does not have to change with most updates.
  • We have the latest version (as of 10 April 2015) of clonezilla on asimov
• Web site: http://clonezilla.org/
• Screen shots
  • http://clonezilla.org/screenshots/?in_path=/00_Clonezilla
• Features
  • Can image and restore both Windows and Linux systems
  • Recover/backup
  • PXE bootable
  • Multicast
  • Supports a wide range of file systems
  • Backup only parts of the disk that are used
• Limitations
  • Cannot restore to smaller disks or partitions

Clonezilla Imaging Notes

• PXEBOOT the node
• Pick Clonezilla_menu_nfs
• Pick Live to create and image
  • After creating a new image run the script /cscf-adm/src/make_clonezilla this will rewrite the PXE menu
  • Images are saved in /coregroup/images
• Pick image to retore an image - WARNING this will REPARTITION AND FORMAT a system WITHOUT ASKING ANY QUESTIONS!!!!!

RIP updates or customizati"> Local RIP updates or customization

• /cscf-adm/src/update_rip 13.7 pxe

USAGE: ./update_rip version  type
   version is numeric
   type is grub,grub2,syslinux,pxe or all
Example: ./update_rip 13.7 grub2
   patches RIP13.7 directory with ghost
Example: ./update_rip all
   updates ALL (pxe,syslinux,grub2) boot images on asimov

• This script will also add cscf specific customizations to RIP
  • It also rewrites the /tftpboot/pxes/pxelinux.cfg/rip_menu and rip_menu13.7 etc

Web Server

• TODO
• Configuration /etc/apache2/conf/local.conf
• See Configuration Example

Web Server Utils

• Source Directory cscf-adm/src.web
• Public Directory cscf-adm/public_html

Httpd server update

• On our server we include /etc/httpd/conf/local.conf from /etc/httpd/conf/httpd.conf for all local settings

Example /etc/httpd/conf/local.conf - permit access to iso images

Alias /iso "/iso"
<Directory /iso>
    order deny,allow
    deny from all
    allow from all
    Options All
</Directory>

• Restart web server: service httpd restart
• Restart mounted iso images: service mount_cds restart

PXE Server

• DNSMASQ Configuration files /cscf-adm/src/dnsmasq - See DNSMASQ
• Home directory /tftpboot/pxes is the PXE root - see PXEBootFilesExample
• Firewall NAT MASQ private network - see CF/NatMasqFirewallExample

Firewall

• Directory /etc/init.d/firewall

     Usage: firewall {start|stop|restart|status|install|uninstall}
     Usage: firewall {add_ssh_host host|del_ssh_host host}
     Usage: firewall {openhost host|blockhost host}
     Usage: firewall {checkhost host}
     

Adding PXE image to the CS Application Server

Notes Outlines how to setup a PXE boot environment for a new system

Installing PXE boot image

• Also see DNSMASQ

/tftpboot/pxes

• PXE root directory
• PXEBootFilesExample

/tftpboot/pxes/pxelinux.cfg/clonezilla_menu_nfs

• Automatically created using /cscf-adm/src/make_clonezilla*

PXE related Applications and Documents

• RIP
  • Recovery is Posible Linux/Windows repair disk
• CSApplicationServer
  • CSCF Application Server Documentation
• CSApplicationServerScripts
  • CSCF Application Server Scripts Documentation
• UbuntuScripts
  • Scripts used in Ubuntu Updates, AD setup, Backup, ETC
• ImageDeployment
  • Image Deployment Documentation
• PXEBootFilesExample
  • PXE Boot Server setup example
• PXELiveBoot
  • Ubuntu PXE Live Boot images - Automatic download and NFS Export update

OLD ASIMOV

*ONLY RELATES TO THE OLD ASIMOV

Historacl reasons for th upgrade

Asimov OLD provides the following services

• Private network nat and firewall for VLAN 420 DNS and DHCP server
• File storage /image server for research and infrastructure
  • For all research and infrastructure Grad desktop images (new and past)
  • Software repository - well over 260 CD/DVD images of installation media
  • VMware images
  • Nexus images
  • Windows Server images for CS
• CS CAD drawings
• Documentation repository
• Site license repository
• Temporary backup space for machines undergoing repair and re-installation
• Deploy studio repository for Macs
• Special projects
• PXE boot server with
  • Live Ubuntu images
  • RIP recovery tools
  • Diagnostic and disk wipe utils
  • All Ubuntu netboot installers - all version
  • Acronis images of machines (see file server)
• Common installation scripts used by research and infrastructure in deploying machines "ubuntu-scripts"

Asimov CSCF Infrastructure Uses:

• DHCP/NAT/Firewall/Network utility server for vlan 420
• PC Imaging in DC (SE Lab, School Pool Laptops and Workstations, RSG workstations/loaner laptops.etc)
• PC Image storage/archival
• PC Image creation/prep (ubuntu-scripts)
• PC PXE Boot Utilities (DBAN, Acronis.etc)
• ISO/Application storage/archival (iso/rawiso/exports) (at least 300 or so current ISOs)
• License storage (site_licenses.txt)
• Macintosh DeployStudio repository for devonshire.cscf (imaging admin workstations, diagnosing Macs.etc)
• Macintosh image storage/archival
• Macintosh package storage/archival

Old Asimov's problems/limitations

• Imaging performance
  • Network/Disk speed is too slow for imaging multiple machines (one machine on our current core image may take upwards of a few hours to complete)
    • Available RAM also plays a factor in this
  • Macs imaging over DeployStudio seem to have speed and timeout issues when transferring large images such as Ed's undergrad image (about 80GB)
• Storage Space
  • We like to backup machines that come in for service before making changes and before redeploying them (incase of future issues) but lack of space means we can't do this for most machine
  • We currently have to store backups on an external USB RAID box due to lack of available drive bays
  • When making new images, we have to shuffle a lot of files around due to the lack of a RAID setup
• Reliability/Expandability
  • Current 'server' has only one desktop power supply installed (one point of failure)
  • No LOM (have to manage console over the remote KVM)
  • No free drive bays for future expansion

Hardware Requirements

• Large hardware RAID in RAID 60 w/ battery backup on controller
  • At least 10TB of usable storage with room for expansion
  • At least 8 drives required, extras configured as hot spares
• 10G SFP+ connection to both vlan 420 and vlan 15
• 16GB+ of RAM
• Redundant hotswap power supplies
• LOM interface
• Hardware warranty
• Ubuntu 14.04.1 LTS server