Example of creating a cluster like cabernet with multiple networks, using a set of virtual machines.

Host VMware cluster setup prerequisites

  • Most of the work was done in salt
    • Given that we were doing all of this in VM's on my desktop we decided to manual create the headnode VM and join it to salt
      • We could have also connected it to a CS salt managed pxe network and bootstrapped the head node install also.

Host hardware interfaces

  • eth0: Connection to campus
  • eth1: Not used here
  • eth2: NIC directly connected to rsg-pc300

Virtual hardware setup

  • Head node: 16.04 Server VM
    • Bridged network interface to host eth0 (outside)
      • Bridge easier for ssh, but salt will work with NAT
    • second interface: host-only network (vmnet3)
      • optionally bridge to host eth2 for rsg-pc300
      • VMWare Virtual Network Editor to set this
    • third interface: host-only network (vmnet2)

  • 2+ identical nodes:
    • Hard drive, but no OS
    • Set PXE first
    • first interface, same host-only as head node's second iface
    • second interface, same as head's third iface

Set up head node

  • hostname cs-tech1-vm, in DNS
  • Start with Ubuntu 16.04 Server
  • Salt bootstrap to salt-rsg-1604.cscf.uwaterloo.ca
  • Accept on salt master
  • salt 'cs-tech1-vm*' state.apply
    • Apply states common to all RSG minions
  • Add head node states to /srv/saltstack/states/top.sls:

    # common.x == states/common/x.sls or x/init.sls
    # Install generic tools for non-LXC machines
    - common.physical
    # Config sshd for remote access and place keys
    - common.ssh
    # /etc/resolv.conf
    - common.networking.dns
    # Package caching
    - common.squid_deb_proxy.server
    # Use our own package cache
    - common.squid_deb_proxy.client
   # Static ifaces for cluster
    - common.networking.cluster_internal
    # sysctl net.ipv4.ip-forward = 1
    # For shorewall NAT
    - common.networking.ip_forward
    # shorewall firewall for NAT
    # Third-party salt formula
    # included via gitfs configured in states/master.d/git_remotes.conf
    - shorewall
    # provides support for dnsmasq for dhcp, dns, pxe boot.
    - common.pxe

  • pull in pillar files in /srv/saltstack/pillar/top.sls:

    # Settings for head node only
    - cs-tech1-vm

  • contents of pillar cs-tech1-vm:

# pillar/cs-tech1-vm/init.sls
# Settings for head node
  # dnsmasq and pxe settings
  - cs-tech1-vm.pxe
  # cluster networking
  - cs-tech1-vm.net
  # export nfs shares
  - cs-tech1-vm.nfs
  # NAT
  - cs-tech1-vm.shorewall
  # ssh keys
  - cs-tech1-vm.ssh

  • contents of pillar cabernet:

# pillar/cabernet/init.sls                 
# Settings for nodes, excluding head node  
  # Turn on DHCP                           
  - cabernet.net                           
  # Mount nfs shares                       
  - cabernet.nfs                           
  # Allow head node's key                  
  - cabernet.ssh                           
  # eth0 interface naming                  
  - cabernet.grub                          

  • Apply changes:
    • salt 'cs-tech1-vm*' state.apply --state-verbose=False test=True
      • Dry-run and suppress 'Clean' returns from already-completed states
    • salt 'cs-tech1-vm*' state.apply --state-verbose=False
      • Same, except real run
      • Will take a few minutes

Install nodes:

  • Due to common.pxe, cs-tech1-vm is now a PXE boot server
  • PXE settings in /srv/saltstack/pillar/cs-tech1-vm/pxe.sls
  • By default, it will send a command to boot local disk
    • pxe:default_pxe: 'bootlocal'
    • To set all nodes to boot, change to, eg, 'pxe-xenial-single-auto'
    • Useful for first cluster setup, but dangerous later
  • To set a node to be installed, edit that entry in the clients list
    • clients list defines MACs, IP's, and boot settings
    • In this case, the eth0 / 10.* entries are the network we use for PXE
    • Add / uncomment preseed: 'pxe-xenial-single-auto'
    • Remember to comment out again after install

  • Apply:
    • Faster to apply only PXE states:
    • salt 'cs-tech1-vm*' common.pxe test=True
    • salt 'cs-tech1-vm*' common.pxe

  • Boot node(s)and wait for Debian installer to start
  • Comment out preseeds again and re-apply as above

Configure nodes

  • Preseed will have installed salt-minion pointing to the same master as the head node
  • Accept node's salt keys on salt master
    • salt-key -l un
    • salt-key -a 'cl1n00*'

  • Define nodegroup:
    • /srv/saltstack/states/master.d/nodegroups.conf
    • cabernet: '*.cabernet_net'
    • Restart salt master to apply
    • systemctl restart salt-master

  • salt -N cabernet test.ping
    • Safely confirm that nodegroup matches what you wanted

  • salt -N cabernet state.apply
    • Nodegroup matching
    • Apply states common to all RSG minions

  • Add states to states/top.sls:

  # Does not include head node!
    # nodegroup defined in states/master.d/nodegroups.conf
    - match: nodegroup
    # Install generic tools for non-LXC machines
    - common.physical
    # Config sshd for remote access and place keys
    - common.ssh
    # /etc/resolv.conf
    - common.networking.dns
    # Use eth0 naming
    - common.grub
    # NFS mounts from head node
    - cluster.nfs_mounts
    # Set up interfaces
    - cluster.dhcp
    # Package caching
    - common.squid_deb_proxy.client                    

  • salt -N cabernet state.apply test=True
    • common.grub will change iface naming to eth# on reboot
    • cluster.dhcp sets eth0 & eth1 ifaces to DHCP
    • Since the reboot applies both at the same time, they work

  • salt -N cabernet state.apply
  • salt -N cabernet system.reboot
    • No return, because it reboots before replying, that's normal

  • salt -N cabernet test.ping
    • Check if they are back up

  • Everything should be done
    • Login and check homedirs, etc

-- NathanFish - 2018-04-26 -- MikeGore - 2018-04-26

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2018-04-26 - MikeGore
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback