UW School of Computer Science UWCS-* Metapackages (Page has been Migrated to Confluence)

https://uwaterloo.atlassian.net/wiki/spaces/CSCFINF/pages/44311019858/Metapackage+Administration

Tue Jun 26, 2018 - https://depot.cs.uwaterloo.ca/ does not answer and I believe it has not answered for some time. That is, HTTPS is not available; http://depot.cs.uwaterloo.ca/ does answer reasonably reliably.

A convenient way to bundle, deploy and update collections of deb packages across UW CS Ubuntu Linux Hosts.

See also:

• https://cs.uwaterloo.ca/cscf/infrastructure/setups/ubuntu/
• https://cs.uwaterloo.ca/cscf/infrastructure/setups/ubuntu/apt/
• RSGMetapackages - RSG MetaPackages documentation for final hand-off to CS for integration into uwcs- metapackages

Metapackage Administration

Modifying Existing Packages

Step 0: Check out packages repository

Existing package definitions are stored in a git repository at git.cs.uwaterloo.ca:/git/a2brenna/packages.git. Clone this repository.

git clone git.cs.uwaterloo.ca:/git/depot.cs.git

Tue Jun 26, 2018 - "git clone" is biased towards the branch defined as current on the server, usually master. -- AdrianPepper Tue Jun 26, 2018 - "git clone --bare" will clone entire repository, with no checked-out branch. -- AdrianPepper Wed Jun 27, 2018 - actually the git trap is as follows... git clone http://git.cs.uwaterloo.ca:/depot.cs cd depot.cs git checkout -b WILY is not the same as... git clone http://git.cs.uwaterloo.ca:/depot.cs cd depot.cs git checkout WILY Also N.B. as shown git clone git.cs.uwaterloo.ca:/git/depot.cs.git also works, and is perhaps preferable since it requires authentication. But that causes us to wonder if the http:// version should be supported. In fact, experiments cause me to wonder whether they are the same repositories. Actually, it appears that the two methods generate radically different .git directories, but that the checked-out contents for equivalent branches are the same.

After changing directory into the repository you must check out the branch corresponding to the release for which you wish to modify packages.

git checkout `lsb_release -sc | tr '[:lower:]' '[:upper:]'`

git checkout TRUSTY

diff -r --brief git-depot.cs-explode/branches/VIVID git-depot.cs/depot.cs | grep -v '[/][.]git[/]'

Step 1: Modify package definition

Debian packages (.deb) have their dependencies listed in a control file. These control files are located within the cloned repository at packages/$PACKAGE_NAME/control/control and are similar to the following...

Package: uwcs-auth
Version: `date +%Y.%m%d%H%M`
Section: admin
Priority: optional
Architecture: all
Depends: libpam-modules, libpam-runtime, libpam-krb5, libpam-ldap, libpam-foreground, openldap-utils, libgssapi-krb5-2, libkrb5-3, krb5-user
Recommends: nscd
Maintainer: CSCF Infrastructure Staff <csi-software@cs.uwaterloo.ca>
Description: Metapackage to install libraries necessary for AD authentication

To modify dependencies or recommendations simply add or remove packages from the appropriate list.

Step 2: Upload the updated package

To upload a new version of an existing package to the repository you must first increment the "Version:" field in the corresponding packages/$PACKAGE_NAME/control/control file. Reprepro will reject your package if you do not, as it does not accept packages of the same name with differing checksums and identical versions.

From within top level of the repository perform the following commands.

cd packages/
../bin/deploy $PACKAGE_NAME

The bin/deploy script uploads the package and runs the appropriate reprepro commands on depot.cs.uwaterloo.ca to publish the new version of the package.

Uploading New Packages

Uploading a new package is nearly identical to modifying an existing package with one exception, you must create a control file in the appropriate place within the git repository. You must also have ssh public key access to depot@depot.cscf.uwaterloo.ca

Step 0: Check out packages repository

Existing package definitions are stored in a git repository at git.cs.uwaterloo.ca:/git/a2brenna/packages.git. Clone this repository.

Step 1: Copy convenient package template

From the top level of the repository...

cp -r packages/template packages/$NEW_PACKAGE_NAME

Step 2: Modify details in template file

It is important to note that it is not possible to have fields which are empty in Debian control files. If a field (such as "Recommends: " for instance) is empty, remove it.

Step 3: Upload package

Identical to Modify Existing Package: Step 2.

Listing of available UWCS Metapackages

For a current listing of the available UWCS Metapackages see: https://cs.uwaterloo.ca/cscf/infrastructure/standard-platforms/architectures#LinuxSupportedApplications

Installing UWCS Metapackages on machines you administrate

Setup apt to use the depot.cs.uwaterloo.ca repository

see https://cs.uwaterloo.ca/cscf/infrastructure/setups/ubuntu/apt/#CSCFdepot

find/choose Package

apt-cache search uwcs-

install pacakage uwcs-<xyz>

apt-get install uwcs-<xyz>

Note some packages may require manual configuration/response the first time they are installed.

Adding a new distro

To support a new distribution, you must add an entry in the distributions configuration file.

Step 0: Append something similar to the following to the depot.cs.uwaterloo.ca:/depot/debian/conf/distributions file, changing the Suite: and Codename: as necessary.

echo "Origin: CSCF
Label: Ubuntu
Suite: precise
Codename: precise
Architectures: alpha amd64 i386 mips mipsel sparc powerpc source
Components: main contrib non-free
Log: precise.log" > /depot/debian/conf/distributions

Step 1: Update the bin/deploy script

Metapackage Server Setup

The metapackage server is a simple, local debian package server called depot.cs.uwaterloo.ca accessible via the web at http://depot.cs.uwaterloo.ca.

Apache Configuration

Depot.cs.uwaterloo.ca uses apache to serve the web interface and to support the use of https:// as a transport mechanism for packages.

Tue Jun 26, 2018 - https://depot.cs.uwaterloo.ca/ does not answer and I believe it has not answered for some time. -- AdrianPepper

Step 0: Install apache

apt-get install --quiet --force-yes apache2

Step 1: Configure apache to serve depot directory

echo "<VirtualHost *:80>
    ServerAdmin a2brenna@csclub.uwaterloo.ca
    ServerName depot.cs.uwaterloo.ca
    ServerAlias depot.cs depot

    DocumentRoot /depot/debian/www

    ErrorLog /var/log/apache2/debian-error.log
    CustomLog /var/log/apache2/debian-access.log combined

    <Directory /depot/debian/www>
        Allow from all
    </Directory>
</virtualHost>" > /etc/apache2/sites-enabled/debian
echo "<VirtualHost *:80>
    ServerAdmin a2brenna@csclub.uwaterloo.ca
    ServerName depot.cs.uwaterloo.ca
    ServerAlias depot.cs depot

    DocumentRoot /depot/debian/www

    ErrorLog /var/log/apache2/debian-error.log
    CustomLog /var/log/apache2/debian-access.log combined

    <Directory /depot/debian/www>
        Allow from all
    </Directory>
</virtualHost>" > /etc/apache2/sites-available/debian
/etc/init.d/apache2 restart

In the near future, these files will by symlinked together as they are identical and should remain so.

Tue Jun 26, 2018 - https://depot.cs.uwaterloo.ca/ does not answer and I believe it has not answered for some time. -- AdrianPepper Tue Jun 26, 2018 - After all, I see no https config there -- AdrianPepper

Reprepro Configuration

We use a utility called reprepro ( http://mirrorer.alioth.debian.org/ ) to manage the repository and enable easy publishing of our various metapackages. Installation, configuration and use are relatively straightforward and are based on the instructions found at http://anonscm.debian.org/gitweb/?p=mirrorer/reprepro.git;a=blob_plain;hb=HEAD;f=docs/short-howto.

Step 0: Create local depot user

It is beneficial to create a local unprivileged user that will own permissions to your repository. It is possible to administrate it as root, but this is less safe, since anyone uploading and maintaining packages will then require root permissions.

adduser depot

To facilitate controlled safe access to this account it is advisable to use ssh keys. At present, access to the depot account (and permissions to manage the repository) are governed by the ssh keys in /home/depot/.ssh/authorized_keys2. Given the small number of administrators at CSCF this solution seems perfectly adequate.

root@depot:~# getent passwd depot
depot:x:1000:1000:CSCF Depot Manager,,,:/home/depot:/bin/bash
root@depot:~# 

Step 1: Install reprepro from your official Ubuntu mirror.

apt-get install --quiet --force-yes reprepro

Step 2: Choose a directory to house your repository.

mkdir -p /depot/debian

Technically this can be an arbitrary empty directory. Depot.cs.uwaterloo.ca:/depot/debian seemed reasonable since we may eventually support distributions that are not debian based, and this naming scheme allows us to encapsulate our entire software packaging solution in a single top level directory, /depot.

Step 3: Create a configuration directory within your repository.

mkdir -p /depot/debian/conf

Step 4: Create a configuration file to define the distributions you support.

echo "Origin: CSCF
Label: Ubuntu
Suite: precise
Codename: precise
Architectures: alpha amd64 i386 mips mipsel sparc powerpc source
Components: main contrib non-free
Log: precise.log" > /depot/debian/conf/distributions

Multiple entries in this file are separated with empty lines.

Since metapackages have no binary components and exist only to conveniently install a set of dependencies, we can safely support numerous architectures. In retrospect, Ubuntu (and CSCF) only supports i386, amd64 and experimental arm architectures, so we can likely safely remove the other entries at some point.

CSCF currently only supports the Ubuntu 12.04 LTS release, codenamed Precise. There is legacy support for Ubuntu 10.10 (Maverick), but this is no longer maintained and should not be used on new machines.

arpepper@u1804github:/tmp/git-depot.cs-explode/repository/depot.cs.git$ git branch -a
  PRECISE
  TRUSTY
  VIVID
  WILY
  XENIAL
* master
arpepper@u1804github:/tmp/git-depot.cs-explode/repository/depot.cs.git$ 

GPG Configuration

Step 0: Install necessary packages

apt-get install --quiet --force-yes gnupg dpkg-sig

Step 1: Generate a key

su - depot
gpg --gen-key

To securely generate a key it is necessary that the machine have sufficient entropy in the entropy pool. This is occasionally a problem on machines that do not have a lot of interactive use. In our case, it took many many days to gather sufficient entropy for the pool in order to generate a proper key.

Step 2: Export public key

su - depot
gpg --armor --export > depot.pub.gpg.key

This public key is what your clients will use to verify the integrity of your signed packages. It is safe to post this somewhere public. In our case CSCF has placed this key at https://cs.uwaterloo.ca/cscf/certs/depot.pub.gpg.key.

Step 3: Enable Signing with reprepro

Add "SignWith: yes" to any distribution entries in /depot/debian/conf/distributions to enable gpg signing of that distributions packages and release data.

"SignWith: yes" is important. It tells reprepro that we'd like to cryptographically sign or packages and release information, ensuring our users (and ourselves) that the packages being installed are legitimate. Properly configuring package signing is subsequently explained. If cryptographically security is not required, you can omit this line and publish unsigned packages, but this will cause the package manager on your client machines to complain every time an installation or upgrade of one of your packages is attempted.

"SignWith: yes" causes reprepro and gnupg to assume that the private signing key is placed in the default reasonable location (where gpg --gen-key placed it). It is possible to use different configuration options to allow reprepro to make more advanced use of gnupg's signing abilities, such as having multiple keys, but this is beyond the scope of CSCF's deployment at this time.

Add package administrator

Adding someone to the set of administrators of the package repository is simple.

SSH key access"> Step 0: SSH key access

Add their public ssh key to /home/depot/.ssh/authorized_keys2.

Add Legato package

IST provides a Debian package of the latest Legato Networker Client. Re-hosting it on depot.cs is convenient.

Step 0: Write script to download package and update our repository

Save the following to /home/depot/bin/legato_update.sh

#!/usr/bin/env bash

set -o errexit
set -o xtrace
set -o nounset

rm /tmp/legato.deb || true
wget -O /tmp/legato.deb http://ist.uwaterloo.ca/download/networker/current/nw_lgtoclnt_amd64.deb

#Will fail if identical version already exists
reprepro -b /depot/debian includedeb trusty /tmp/legato.deb || true

Step 1: Make sure user depot has cronjob to run script

Include the following line in user depot's crontab file

0 0 * * * ~/bin/legato_update.sh

root@depot:~# apt-cache showpkg reprepro | head -4
Package: reprepro
Versions: 
4.8.2-1ubuntu0.1 (/var/lib/apt/lists/mirror.csclub.uwaterloo.ca_ubuntu_dists_precise-updates_universe_binary-amd64_Packages) (/var/lib/dpkg/status)
 Description Language: 
root@depot:~# 

arpepper@u1804github:~$ apt-cache showpkg reprepro | head -4
Package: reprepro
Versions: 
5.1.1-1 (/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_Packages) (/var/lib/apt/lists/mirror.csclub.uwaterloo.ca_ubuntu_dists_bionic_universe_binary-amd64_Packages)
 Description Language: 
arpepper@u1804github:~$

-- AnthonyBrennan - 2013-05-22
-- AdrianPepper - 2018-06-26

git checkout `lsb_release -sc | tr '[:lower:]' '[:upper:]'`

git checkout TRUSTY