ClusterExample < CF

CF Web>Infrastructure>SaltStack>SaltStackCSCF>ClusterExample (2018-04-26, MikeGore) (raw view)
---++ Purpose
Example of creating a cluster like cabernet with multiple networks, using a set of virtual machines.

---++ Host VMware cluster setup prerequisites 
   * Most of the work was done in salt 
      * Given that we were doing all of this in VM's on my desktop we decided to manual create the headnode VM and join it to salt 
         * We could have also connected it to a CS salt managed pxe network and bootstrapped the head node install also.

  
---++ Host hardware interfaces
   * eth0: Connection to campus
   * eth1: Not used here
   * eth2: NIC directly connected to rsg-pc300

---++ Virtual hardware setup

   * Head node: 16.04 Server VM
      * Bridged network interface to host eth0 (outside)
         * Bridge easier for ssh, but salt will work with NAT
      * second interface: host-only network (vmnet3)
         * optionally bridge to host eth2 for rsg-pc300
         * VMWare Virtual Network Editor to set this
      * third interface: host-only network (vmnet2)

   * 2+ identical nodes:
      * Hard drive, but no OS
      * Set PXE first
      * first interface, same host-only as head node's second iface
      * second interface, same as head's third iface 

---++ Set up head node

   * hostname cs-tech1-vm, in DNS
   * Start with Ubuntu 16.04 Server
   * Salt bootstrap to salt-rsg-1604.cscf.uwaterloo.ca
   * Accept on salt master
   * salt 'cs-tech1-vm*' state.apply
      * Apply states common to all RSG minions
   * Add head node states to =/srv/saltstack/states/top.sls=:

<verbatim>
  'cs-tech1-vm.cs.uwaterloo.ca':
    # common.x == states/common/x.sls or x/init.sls
    # Install generic tools for non-LXC machines
    - common.physical
    # Config sshd for remote access and place keys
    - common.ssh
    # /etc/resolv.conf
    - common.networking.dns
    # Package caching
    - common.squid_deb_proxy.server
    # Use our own package cache
    - common.squid_deb_proxy.client
   # Static ifaces for cluster
    - common.networking.cluster_internal
    # sysctl net.ipv4.ip-forward = 1
    # For shorewall NAT
    - common.networking.ip_forward
    # shorewall firewall for NAT
    # Third-party salt formula
    # included via gitfs configured in states/master.d/git_remotes.conf
    - shorewall
    # provides support for dnsmasq for dhcp, dns, pxe boot.
    - common.pxe
</verbatim>

   * pull in pillar files in =/srv/saltstack/pillar/top.sls=:

<verbatim>
  'cs-tech1-vm.cs.uwaterloo.ca':
    # Settings for head node only
    - cs-tech1-vm
</verbatim>

   * contents of pillar cs-tech1-vm:

<verbatim>
# pillar/cs-tech1-vm/init.sls
# Settings for head node
include:
  # dnsmasq and pxe settings
  - cs-tech1-vm.pxe
  # cluster networking
  - cs-tech1-vm.net
  # export nfs shares
  - cs-tech1-vm.nfs
  # NAT
  - cs-tech1-vm.shorewall
  # ssh keys
  - cs-tech1-vm.ssh
</verbatim>

   * contents of pillar cabernet:

<verbatim>
# pillar/cabernet/init.sls                 
# Settings for nodes, excluding head node  
include:                                   
  # Turn on DHCP                           
  - cabernet.net                           
  # Mount nfs shares                       
  - cabernet.nfs                           
  # Allow head node's key                  
  - cabernet.ssh                           
  # eth0 interface naming                  
  - cabernet.grub                          
</verbatim>

   * contents of all these files available in git
      * https://git.uwaterloo.ca/salt_cs/salt_rsg_pillar/tree/master

   * Apply changes:
      * =salt 'cs-tech1-vm*' state.apply --state-verbose=False test=True=
         * Dry-run and suppress 'Clean' returns from already-completed states
      * =salt 'cs-tech1-vm*' state.apply --state-verbose=False=
         * Same, except real run
         * Will take a few minutes

---++ Install nodes:

   * Due to =common.pxe=, =cs-tech1-vm= is now a PXE boot server
   * PXE settings in =/srv/saltstack/pillar/cs-tech1-vm/pxe.sls=
   * By default, it will send a command to boot local disk
      * =pxe:default_pxe: 'bootlocal'=
      * To set all nodes to boot, change to, eg, 'pxe-xenial-single-auto'
      * Useful for first cluster setup, but dangerous later
   * To set a node to be installed, edit that entry in the =clients= list
      * =clients= list defines MACs, IP's, and boot settings
      * In this case, the eth0 / 10.* entries are the network we use for PXE 
      * Add / uncomment =preseed: 'pxe-xenial-single-auto'=
      * Remember to comment out again after install

   * Apply:
      * Faster to apply only PXE states:
      * =salt 'cs-tech1-vm*' common.pxe test=True=
      * =salt 'cs-tech1-vm*' common.pxe=

   * Boot node(s)and wait for Debian installer to start
   * Comment out preseeds again and re-apply as above 

---++ Configure nodes

   * Preseed will have installed salt-minion pointing to the same master as the head node
   * Accept node's salt keys on salt master
      * =salt-key -l un=
      * =salt-key -a 'cl1n00*'=

   * Define nodegroup:
      * =/srv/saltstack/states/master.d/nodegroups.conf=
      * =cabernet: '*.cabernet_net'=
      * Restart salt master to apply
      * =systemctl restart salt-master=

   * =salt -N cabernet test.ping=
      * Safely confirm that nodegroup matches what you wanted

   * =salt -N cabernet state.apply=
      * Nodegroup matching
      * Apply states common to all RSG minions

   * Add states to =states/top.sls=:

<verbatim>
  # Does not include head node!
  'cabernet':
    # nodegroup defined in states/master.d/nodegroups.conf
    - match: nodegroup
    # Install generic tools for non-LXC machines
    - common.physical
    # Config sshd for remote access and place keys
    - common.ssh
    # /etc/resolv.conf
    - common.networking.dns
    # Use eth0 naming
    - common.grub
    # NFS mounts from head node
    - cluster.nfs_mounts
    # Set up interfaces
    - cluster.dhcp
    # Package caching
    - common.squid_deb_proxy.client                    
</verbatim>

   * =salt -N cabernet state.apply test=True=
      * common.grub will change iface naming to eth# on reboot
      * cluster.dhcp sets eth0 & eth1 ifaces to DHCP
      * Since the reboot applies both at the same time, they work 

   * =salt -N cabernet state.apply=
   * =salt -N cabernet system.reboot=
      * No return, because it reboots before replying, that's normal

   * =salt -N cabernet test.ping=
      * Check if they are back up

   * Everything should be done
      * Login and check homedirs, etc

-- Main.NathanFish - 2018-04-26
-- Main.MikeGore - 2018-04-26
Topic revision: r2 - 2018-04-26 - MikeGore
Information in this area is meant for use by CSCF staff and is not official documentation, but anybody who is interested is welcome to use it if they find it useful.
Other Webs
My links
- People
- CERAS
- WatForm
- Tetherless lab
- Ubuntu Main.HowTo
- eDocs
- RGG NE notes
- RGG
- CS infrastructure
- Grad images
Edit