CSCF Nettop Deployment

An overview of the various technologies at work providing service to MC3022, with links to further documentation.

When A Deployed Nettop Boots

Where Is Everything Located?

1. Terminal hostname and IP address acquired via DHCP service running on the following two boot servers.
   • bs202.cscf.uwaterloo.ca: An Ubuntu LXC guest system on dc-3558-208.storage.cs.uwaterloo.ca
   • bs204.cscf.uwaterloo.ca: An Ubuntu LXC guest system on mc-3015-208.cloud.cs.uwaterloo.ca
   • Both DHCP servers are configured by the following files which are synced between the two servers.
     • /etc/dhcp/dhcp.conf
     • /etc/dhcp/csnets.conf - All Nettop DHCP info in this file.
2. Terminal OS kernel acquired via TFTP service also running on above listed boot servers.
   • Exact kernel location configured in /etc/dhcp/csnets.conf for each Nettop entry. For most Nettops this is one of two paths.
     • /srv/tftp/thin_client/mc/bionicbeaver/pxelinux.0
     • /srv/tftp/thin_client/dc/bionicbeaver/pxelinux.0
3. Network location and access protocol for full terminal OS acquired from kernel configuration files on the above listed boot servers.
   • • /srv/tftp/thin_client/mc/bionicbeaver/pxelinux.cfg/default
     • /srv/tftp/thin_client/dc/bionicbeaver/pxelinux.cfg/default
4. Full terminal OS mounted via NFS from mc-3015-nfsgateway.cs.uwaterloo.ca (172.19.167.5):
   • Nettop root filesystem for the CS-TEACHING environment in MC
     • Currently, mc-3015-nfsgateway.cs.uwaterloo.ca:/fsys/nettop_os_deployment/TEACHING/bionicbeaver-upt-aug-2019
   • Nettop root filesystem for the CS-GENERAL environment in DC
     • Currently, mc-3015-nfsgateway.cs.uwaterloo.ca:/fsys/nettop_os_deployment/GENERAL/bionicbeaver-upt-aug-2019
5. User home directory (networked) disk space mounted via Kerberized NFS as defined in the following file on a local Nettop.
   • /etc/fstab

Acquiring Nettop OS Image

The boot process for all CSCF managed Nettop terminals involves two servers - a boot_server and an image_server - three network protocols - DHCP, TFTP and NFS - and one level of indirection. This process is complicated but it works.

All CSCF Nettop terminals are configured to boot using PXE over the network. They make a DHCP request which is answered by one of three identically configured boot_servers.

• bs202.cscf.uwaterloo.ca: An Ubuntu LXC guest system on dc-3558-208.storage.cs.uwaterloo.ca.
• bs204.cscf.uwaterloo.ca: An Ubuntu LXC guest system on mc-3015-208.cloud.cs.uwaterloo.ca

The boot_servers provide two network services for the Nettops

• DHCP: For establishing terminal networking.
  • Configured in boot_server:/etc/dhcp/dhcp.conf with specific terminal configuration included in boot_server:/etc/dhcp/csnets.conf
• TFTP: For initiating the download of the Nettop OS image.
  • Configured in boot_server:/etc/inetd.conf

Based upon a booting Nettop's ethernet (MAC) address, the DHCP service will supply the terminal's IP address, DNS domain and a TFTP path to the OS image. Below is an example of the DHCP configuration for a typical CSCF Nettop terminal as found in boot_server:/etc/dhcp/csnets.conf.

...
        host nettop31.student.cs.uwaterloo.ca {
                option host-name "nettop31";
                option domain-name "student.cs.uwaterloo.ca";
                option domain-search "uwaterloo.ca student.cs.uwaterloo.ca";
                hardware ethernet b8:ae:ed:7f:70:23;
                fixed-address 172.19.50.101;
                filename "thin_client/mc/bionicbeaver/pxelinux.0";
        }
...

Because no next-server option is specified in any boot_server's dhcp.conf file, a booting Nettop terminal assumes that its next-server is the same as the boot_server. This is the host from which a PXE client downloads its bootloader, kernel and initramfs using TFTP. The filename option in the DHCP example above specifies the TFTP path to the linux kernel to download - the name "bionicbeaver" being the current Ubuntu OS version for the CSCF Nettop terminals. This kernel is executed on the Nettop with command-line parameters passed to it by the bootloader. These parameters are specified within the TFTP service in the file boot_server:/srv/tftp/thin_client/mc/bionicbeaver/pxelinux.cfg/default, the contents of which are as follows.

default linux
label linux
 kernel images/amd64/vmlinuz
 append initrd=images/amd64/initrd.img root=/dev/nfs nfsroot=172.19.167.5:/fsys/nettop_os_deployment/TEACHING/bionicbeaver-upt-aug-2019,udp,rsize=1048576,wsize=1048576,acregmin=1200,acdirmin=1200,acregmax=1200,acdirmax=1200 ip=dhcp rw overlay

This is the point where the process of loading a Nettop's OS switches from TFTP loading to NFS loading and also switches from the boot_server to the image_server - one level of indirection. The downloaded kernel loads initramfs - which is built with an NFS client - and now operates independent from the boot_server. The path to the NFS mounted Nettop OS image (nfsroot option) was passed to the kernel as a parameter at boot time and is used by initramfs:
172.19.167.5:/fsys/nettop_os_deployment/TEACHING/bionicbeaver-upt-aug-2019

The IP address 172.19.167.5 represents the Nettop image_server used by CSCF.

The OS image is downloaded and cached on the Nettop where it can operate independent of the image source.

Mounting Networked File Systems Including User Home Directories

For the time being, configuration files unique to a specific Nettop terminal (keytab, root crontabs, etc.) continue to be stored and mounted from either nfs-gateway-202.cscf or nfs-gateway-204.cscf. These will be migrated to the OS image server at a later date.

CSCF managed Nettop terminals have a customized system service named homedirs.service which has been added to the /etc/systemd/system directory of the Nettop OS image. Its purpose is to run a CSCF maintained custom script named cscf-homedirs.sh found in the /usr/bin directory of the OS image. This bash script manages the mounting of network filesystems to the local Nettop.

1. NFS mount of the Nettop terminal's overlays. Unique configuration files (keytab, root crontabs, etc.) also maintained on the OS image_server.
   • The overlay is mounted on the /nfs directory of the local Nettop.
     eg. 10.15.154.152:/srv/nfs/172.19.50.100 937724928 90962944 842848256 10% /nfs
   • The Nettop OS image then utilizes symbolic links to integrate files in the overlay.
     eg. lrwxrwxrwx 1 root root 20 Aug 19 2015 /etc/krb5.keytab -> /nfs/etc/krb5.keytab
2. Kerberized NFS mount of networked user file system from regional NetApp server as defined in the /etc/fstab of the OS image.
   • Kerberized NFS mounts are dependent upon the Nettop terminal's keytab file being accessible so mounting of NetApp shares must occur after the mounting of the overlays.

Physical Hardware

There are currently a three linux container in the teaching environment dedicated to supporting MC3022 and MC3018 and MC2061. They should all be configured identically.

• bs202.cscf.uwaterloo.ca
• bs204.cscf.uwaterloo.ca

There is one NFS server providing the Nettop root filesystem for both CS-TEACHING and CS-GENERAL environments. * mc-3015-nfsgateway.cscf.uwaterloo.ca

There is one NFS server providing the Nettop root filesystem for the CS-TEACHING environment in MC. * nfs-gateway-202.cscf.uwaterloo.ca

There is one NFS server providing the Nettop root filesystem for the CS-GENERAL environment in DC. * nfs-gateway-204.cscf.uwaterloo.ca

Boot

The Nettop clients use PXE to boot over the network. The BIOS on the client is configured to boot only using PXE.

There are two server components used to facilitate booting

• A DHCP server
• A TFTP server

The DHCP server provides a unique IP address to each Nettop by MAC address. The DHCP server also informs each client that its "next-server" is itself (by IP address). The "next-server" is the host from which a PXE boot client will attempt to download its bootloader, kernel and initramfs.

The TFTP server is configured to use inetd and atftpd to serve the syslinux bootloader, linux kernel and initramfs. In our case, the TFTP server is also hosted on the same server as the DHCP service, but it is not strictly necessary to have the TFTP server on the same machine as the DHCP server.

Once the kernel has been downloaded, it is executed with command-line parameters passed to it by the bootloader. These command line parameters are specified on the tftp server in the file /srv/tftp/$OS_RELEASE/pxelinux.cfg/default. From this point the kernel loads the initramfs and can theoretically operate independent from the servers. The path to the NFS mounted root filesystem is passed to the kernel as a parameter at boot time.

Other important non-standard boot parameters include, 'rw', and 'overlay'. Ubuntu does not function well when the kernel is explicitly told to mount the root filesystem in read only mode. In our system, the NFS server enforces read only behaviour, so making the client happy by passing this parameter is harmless. The custom initramfs hook script that mounts the writable tmpfs overlay does so only if the parameter 'overlay' is passed to the system at boot time. Without it, the client will find itself unable to write to its root filesystem and many things simply will not work.

Filesystem Image

The root filesystem used by the Nettops is served over NFS from /fsys/nettop_os_deployment// from the NFS server. Any changes made to this directory on the server will be reflected on all Nettops immediately.

Authentication

Preparing a Nettop Server

A Nettop Server requires a few pieces of software to function

• DHCP Server (dhcp3-server)
• INETD (openbsd-inetd)
• TFTP Server (atftpd)
• NFS Server (nfs-kernel-server or unfs3 (for LXC Containers))
• (please list commands one would type to install theses packages. See http://www.cs.uwaterloo.ca/cscf/software/ubuntu/UW-CS-Workstation/ as an example)

Configuring DHCP

The /etc/dhcp3/dhcpd.conf file (or its equivalent) will require the following options.

• option domain-name "$DOMAIN";
• option domain-name-servers $DNS1,$DNS2;
• allow booting;
• allow bootp;

The configuration file will also require a correct subnet definition for its VLAN.

$DOMAIN will be student.cs.uwaterloo.ca for the CS-TEACHING environment, or cs.uwaterloo.ca for the CS-GENERAL environment.

$DNS1,$DNS2 etc are the IP addresses of the correct DNS servers.

Configuring INETD

The /etc/inetd.conf file (or its equivalent) will require the following.

• tftp dgram udp4 wait nobody /usr/sbin/tcpd /usr/sbin/in.tftpd --tftpd-timeout 300 --retry-timeout 5 --mcast-port 1758 --mcast-addr 239.239.239.0-255 --mcast-ttl 1 --maxthread 100 --verbose=5 /srv/tftp

/srv/tftp is whatever directory the PXE boot images are being served from.

Configuring NFS

NFS should be configured to serve the directory containing the filesystem image to the entire subnet READ ONLY and without root squashing. For example...

• /srv/nfs/test_nettop_root 129.97.59.128/255.255.255.128(ro,no_root_squash,no_subtree_check)

Individual directories per client must also be served RW (for storing unique Kerberos keytab files). It is necessary to name these directories the IP address of the corresponding client. For example...

• /srv/nfs/129.97.59.177 129.97.59.177(rw,no_root_squash,no_subtree_check)

It is best to create a directory per IP Address in the subnet when you're setting up the server so that no further modifications to NFS are necessary when adding new clients. Create a writable directory for its IP address in /srv/nfs/$IP. This directory must include an empty file at /srv/nfs/$IP/etc/krb5.keytab. This allows Clayton's scripts to correctly generate and place a keytab file to allow Active Directory authentication.

mkdir -pv /srv/nfs/$IP/etc && touch /srv/nfs/$IP/etc/krb5.keytab

Add the necessary entry to export /srv/nfs/$IP to the correct $IP in /etc/exports

echo "/srv/nfs/$IP $IP(rw,no_root_squash,no_subtree_check)" >> /etc/exports

Preparing a Filesystem Image

THIS SECTION IS UNDER REVISION AS BUILD FORMULA HAS CHANGED FOR UBUNTU 18.04 - See RT#950077 for up to date details.

The easiest way to create an Ubuntu (or Debian) filesystem image is to use debootstrap, provided by the debootstrap package. For example...

• debootstrap --arch $ARCH $RELEASE $FOO $MIRROR

This command will create a minimal Ubuntu (or Debian) installation in the directory $FOO.

$ARCH is the binary architecture (usually amd64 or i686).

$RELEASE is the name of the release (maverick, oneiric, natty etc.)

$MIRROR is the URL of the Debian compatible mirror in question (for our purposes usually http://mirror.csclub.uwaterloo.ca/ubuntu)

After installing this minimal system, it is necessary to install additional packages and edit some configuration files. This can be accomplished by chrooting into the filesystem image and installing the necessary packages. It is necessary to install a working kernel.

In order to facilitate booting and provide a writable overlay, several scripts must be added to the ubuntu boot process.

/etc/init/cscf-keytab.conf is an upstart script that mounts the NFS directory containing the host specific keytab file. template file

/etc/init/cscf-swap.conf turns any attached hard disks into encrypted swap space to relieve memory pressure. template file

/etc/initramfs-tools/scripts/nfs-bottom/writable_overlay modifies the initramfs to mount a writable overlay filesystem on / before handing off execution to /sbin/init. template file

Additionally /etc/initramfs-tools/initramfs.conf requires the following option(s).

• BOOT=nfs

Additionally /etc/initramfs-tools/modules file must include a line for aufs so the aufs module is included in the initramfs.

CSCF uses a custom chsh script. It is important to replace the binary /usr/bin/chsh with this script.

We configure /etc/lightdm/lightdm.conf to disable the Guest account and User List.

[SeatDefaults]
greeter-show-manual-login=true
greeter-hide-users=true
allow-guest=false

Differences between CS-GENERAL and CS-TEACHING images

The following files have important differences in them, if you're migrating an image between environments be sure to configure them correctly for your target environment.

• /etc/fstab
• /etc/krb5.conf
• /etc/ldap.conf
• /etc/ldap/ldap.conf
• /etc/resolv.conf

Preparing a PXE Boot Image

This Linux Journal article is very comprehensive and helpful. Read it if you have time. Otherwise, shortcuts below.

While chrooted into the filesystem image, after configuring /etc/initramfs-tools, generate a new initramfs

• update-initramfs -uk all

From one of the existing boot servers (bs004.student.cs.uwaterloo.ca, bs102.cs.uwaterloo.ca), copy /srv/tftp to your new Nettop Server. Replace the existing kernel and initramfs in /usrv/tftp/$RELEASE/image/$ARCH with the ones you've just generated. It may be necessary to change /srv/tftp/$FOO/pxelinux.cfg/default to point at the correct new kernel and initramfs.

/srv/tftp/$FOO/pxelinux.cfg/default must contain an append entry for each label with the following.

• append initrd=images/amd64/wheezy/initrd.img root=/dev/nfs nfsroot=129.97.59.190:/srv/nfs/wheezy ip=dhcp rw overlay nomodeset

nomodeset avoids some graphics bugs. In future it might be wise to remove this.

nfsroot has to use the IP address of the Nettop Server

rw is necessary (for the client to think it has write access while being denied writes by the server).

Adding a Nettop Client

Adding a new client is a two step process.

Create an appropriate inventory record including hostname, domain, ip address and mac address.

On linux.cscf.uwaterloo.ca, in /root/pxectl, update the hosts.py file to include your new host(s) with a supported target.

On linux.cscf.uwaterloo.ca, in /root/pxectl, run push.sh.

Normally NFS traffic between the Nettop VLAN in MC and the NFS Server in MC is blocked by the firewall. In order for Nettops to work correctly each new host must be added to the firewall rule allowing NFS traffic between itself and the MC Cloud hosts (mc-cloud002-n{1,2,3,4,5},cloud.cs.uwaterloo.ca).

Adding a package to a Nettop Deployment

If you wish to add a package to the MC CS-TEACHING environment, run the following commands on nfs-gateway-202.cscf.uwaterloo.ca.

chroot /srv/nfs/$RELEASE
apt-get install $PACKAGE

If you wish to add a package to the DC CS-GENERAL environment, run the following commands on nfs-gateway-204.cscf.uwaterloo.ca.

chroot /srv/nfs/$RELEASE
apt-get install $PACKAGE

Troubleshooting Nettop Issues

The most common Nettop issue is refusing to boot. Most often this is accompanied by complaints of "Missing boot media", or an inability to establish a network link. In this case, ensure physical network connectivity and check to see that the port is on the correct vlan. If the Nettop fails to boot due to DHCP timeout, check the configuration on the relevent DHCP server (bs202.cscf.uwaterloo.ca, bs004.student.cs.uwaterloo.ca, bs102.cs.uwaterloo.ca) to ensure that an entry exists for the particular Nettop.

-- AnthonyBrennan - 06 Oct 2011