Purpose

Host VMware cluster setup prerequisites

• Most of the work was done in salt
  • Given that we were doing all of this in VM's on my desktop we decided to manual create the headnode VM and join it to salt
    • We could have also connected it to a CS salt managed pxe network and bootstrapped the head node install also.

Host hardware interfaces

• eth0: Connection to campus
• eth1: Not used here
• eth2: NIC directly connected to rsg-pc300

Virtual hardware setup

• Head node: 16.04 Server VM
  • Bridged network interface to host eth0 (outside)
    • Bridge easier for ssh, but salt will work with NAT
  • second interface: host-only network (vmnet3)
    • optionally bridge to host eth2 for rsg-pc300
    • VMWare Virtual Network Editor to set this
  • third interface: host-only network (vmnet2)

• 2+ identical nodes:
  • Hard drive, but no OS
  • Set PXE first
  • first interface, same host-only as head node's second iface
  • second interface, same as head's third iface

Set up head node

• hostname cs-tech1-vm, in DNS
• Start with Ubuntu 16.04 Server
• Salt bootstrap to salt-rsg-1604.cscf.uwaterloo.ca
• Accept on salt master
• salt 'cs-tech1-vm*' state.apply
  • Apply states common to all RSG minions
• Add head node states to /srv/saltstack/states/top.sls:

  'cs-tech1-vm.cs.uwaterloo.ca':
    # common.x == states/common/x.sls or x/init.sls
    # Install generic tools for non-LXC machines
    - common.physical
    # Config sshd for remote access and place keys
    - common.ssh
    # /etc/resolv.conf
    - common.networking.dns
    # Package caching
    - common.squid_deb_proxy.server
    # Use our own package cache
    - common.squid_deb_proxy.client
   # Static ifaces for cluster
    - common.networking.cluster_internal
    # sysctl net.ipv4.ip-forward = 1
    # For shorewall NAT
    - common.networking.ip_forward
    # shorewall firewall for NAT
    # Third-party salt formula
    # included via gitfs configured in states/master.d/git_remotes.conf
    - shorewall
    # provides support for dnsmasq for dhcp, dns, pxe boot.
    - common.pxe

• pull in pillar files in /srv/saltstack/pillar/top.sls:

  'cs-tech1-vm.cs.uwaterloo.ca':
    # Settings for head node only
    - cs-tech1-vm

• contents of pillar cs-tech1-vm:

# pillar/cs-tech1-vm/init.sls
# Settings for head node
include:
  # dnsmasq and pxe settings
  - cs-tech1-vm.pxe
  # cluster networking
  - cs-tech1-vm.net
  # export nfs shares
  - cs-tech1-vm.nfs
  # NAT
  - cs-tech1-vm.shorewall
  # ssh keys
  - cs-tech1-vm.ssh

• contents of pillar cabernet:

# pillar/cabernet/init.sls                 
# Settings for nodes, excluding head node  
include:                                   
  # Turn on DHCP                           
  - cabernet.net                           
  # Mount nfs shares                       
  - cabernet.nfs                           
  # Allow head node's key                  
  - cabernet.ssh                           
  # eth0 interface naming                  
  - cabernet.grub                          

• contents of all these files available in git
  • https://git.uwaterloo.ca/salt_cs/salt_rsg_pillar/tree/master

• Apply changes:
  • salt 'cs-tech1-vm*' state.apply --state-verbose=False test=True
    • Dry-run and suppress 'Clean' returns from already-completed states
  • salt 'cs-tech1-vm*' state.apply --state-verbose=False
    • Same, except real run
    • Will take a few minutes

Install nodes:

• Due to common.pxe, cs-tech1-vm is now a PXE boot server
• PXE settings in /srv/saltstack/pillar/cs-tech1-vm/pxe.sls
• By default, it will send a command to boot local disk
  • pxe:default_pxe: 'bootlocal'
  • To set all nodes to boot, change to, eg, 'pxe-xenial-single-auto'
  • Useful for first cluster setup, but dangerous later
• To set a node to be installed, edit that entry in the clients list
  • clients list defines MACs, IP's, and boot settings
  • In this case, the eth0 / 10.* entries are the network we use for PXE
  • Add / uncomment preseed: 'pxe-xenial-single-auto'
  • Remember to comment out again after install

• Apply:
  • Faster to apply only PXE states:
  • salt 'cs-tech1-vm*' common.pxe test=True
  • salt 'cs-tech1-vm*' common.pxe

• Boot node(s)and wait for Debian installer to start
• Comment out preseeds again and re-apply as above

Configure nodes

• Preseed will have installed salt-minion pointing to the same master as the head node
• Accept node's salt keys on salt master
  • salt-key -l un
  • salt-key -a 'cl1n00*'

• Define nodegroup:
  • /srv/saltstack/states/master.d/nodegroups.conf
  • cabernet: '*.cabernet_net'
  • Restart salt master to apply
  • systemctl restart salt-master

• salt -N cabernet test.ping
  • Safely confirm that nodegroup matches what you wanted

• salt -N cabernet state.apply
  • Nodegroup matching
  • Apply states common to all RSG minions

• Add states to states/top.sls:

  # Does not include head node!
  'cabernet':
    # nodegroup defined in states/master.d/nodegroups.conf
    - match: nodegroup
    # Install generic tools for non-LXC machines
    - common.physical
    # Config sshd for remote access and place keys
    - common.ssh
    # /etc/resolv.conf
    - common.networking.dns
    # Use eth0 naming
    - common.grub
    # NFS mounts from head node
    - cluster.nfs_mounts
    # Set up interfaces
    - cluster.dhcp
    # Package caching
    - common.squid_deb_proxy.client                    

• salt -N cabernet state.apply test=True
  • common.grub will change iface naming to eth# on reboot
  • cluster.dhcp sets eth0 & eth1 ifaces to DHCP
  • Since the reboot applies both at the same time, they work

• salt -N cabernet state.apply
• salt -N cabernet system.reboot
  • No return, because it reboots before replying, that's normal

• salt -N cabernet test.ping
  • Check if they are back up

• Everything should be done
  • Login and check homedirs, etc

-- NathanFish - 2018-04-26 -- MikeGore - 2018-04-26