ST E-mail Gateway

Summary

There are two email gateways into ST.

The first is quite simple. Mail sent to either of

ST-comment@math.uwaterloo.ca
ST-comment@cs.uwaterloo.ca

that has a Subject of the form

[UW-MFCF #ITEM_NUMBER] the real subject

becomes a comment in ST #ITEM_NUMBER, recorded with just the real subject. The subject prefix can be omitted by mailing to either of

ST-comment+ITEM_NUMBER@math.uwaterloo.ca
ST-comment+ITEM_NUMBER@cs.uwaterloo.ca

It moves each message attachment into

https://www.cs.uwaterloo.ca/cscf/internal/edocs/rtattach/ITEM_NUMBER/attachment_name

replacing it in the message with a URL, after the line "Attachments:" appended to the end of the message.

The second gateway does more. Mail sent to

rt-em-gw+ITEM_NUMBER@cs.uwaterloo.ca

is preprocessed before being resent to the simple gateway above. It attempts to remove included mail messages, by replacing lines that begin with > with the text > [snip...]. This has the advantage of avoiding those conversational mail messages that just keep growing. It has the disadvantage that it can lose included mail messages that were previously recorded in the item.

Details of this second gateway follow.

Introduction

The ST E-mail Gateway provides a mechanism for sending e-mail messages directly to the Service Tracking (ST) system. Each message is directed at a specific ST item, and shows up as a comment added to that item. That is, sending a message to the ST system with a given ST item number is equivalent to adding the message contents in the comment box of that item, and then clicking the "Update" button.

The implementation is based upon a feature of the ST system, which adds messages sent to ST-comment@cs with [UW-MFCF #STNUM] as the subject (where STNUM is the number of the ST item) as comments to that item.

In addition, any attachments in that email are automatically extracted, saved as discrete files in CS web space ("eDocs") and then replaced with a link to those documents in the resulting ST entry. This is particularly convenient for forwarding Purchase Orders, quotes and other documents that may be relevant to a given request.

Usage

To send a message to the ST system, simply e-mail rt-em-gw+[STNUM]@cs.uwaterloo.ca, where [STNUM] is the number of the ST item. For example, to add content to ST #58225, send a message to rt-em-gw+58225@cs.uwaterloo.ca. This address can appear in either the To or CC fields of the e-mail header, but not in the BCC field.

Implementation

The rt-em-gw User

A special user had been added to the system, named rt-em-gw. This allows us to have an address to send e-mails to. The gateway implementation resides entirely under this user's home directory, which is /u2/rt-em-gw in the CS general computing environment. This directory contains the following files and sub-directories of interest:

• ~/.procmailrc Directs incoming e-mails to the gateway script
• ~/emailgw/ The main implementation directory
• ~/emailgw/bin/emailgw.pl The gateway implementation, as a Perl script
• ~/emailgw/lib/ Perl modules required by the gateway script, but unavailable in the default CS Perl installation
• ~/emailgw/log Log file used by the gateway script
• ~/emailgw/plugins/ Plug-in scripts that can be used to process a message before it is forwarded to the ST system

Procmail Setup

We use Procmail to direct any message containing rt-em-gw in either the To or CC fields of the header to further processing by the gateway script. This is done by the following .procmailrc file in the rt-em-gw home directory:

MAILDIR=$HOME/mail

:0:
* ^(To|Cc).*rt-em-gw\+
| $HOME/emailgw/bin/emailgw.pl >> $HOME/emailgw/log

The Gateway Script

The gateway is implemented as a Perl script. A detailed documentation is provided in-line. Here we only give a brief outline of the way the script works:

1. A message is read from standard input
2. The To and CC fields are searched for an address of the expected form (i.e., rt-em-gw+[STNUM]@cs.uwaterloo.ca)
3. Looks for an existing eDoc directory for the specified ST item, and creates one if necessary (an eDoc directory holds attachments relevant to the item)
4. Creates a MIME parser for the message (required for extracting message parts) and parses the message
5. Creates a new mail message, with ST-comment@math as the recipient and [UW-MFCF #STNUM] as the subject
6. If the message is composed of multiple parts, converts it to a text-only message: text parts are joined in-line, and binary parts saved under the relevant eDoc directory and appear as HTML links
7. Applies plug-ins from the ~/emailgw/plugins/ directory (see below)
8. Removes the special rt-em-gw address from the CC field, so that it doesn't show up in the result (this is just for aesthetic reasons)
9. Prepends text to the message, containing the original header fields (Subject, To, Date, etc.), so that they show up in the ST comments (the built-in ST fields show the information of the message constructed by the gateway)
10. Sends the message

The source code of the script can be viewed here.

Plug-ins

Plug-ins provide an easy way for performing text manipulation on a message before it is sent to the ST system. Two examples currently implemented include a plug-in to remove the original text from a response message (i.e., lines starting with the greater-than sign), and for removing a GPG signature.

A plug-in is written as a Perl script, with one predefined variable named $body$, which contains the message's body text. A plug-in is expected to modify this variable in-place (i.e., no value needs to be returned from the script). As an example, consider the following plug-in to remove the original text from a response message:

# Get an array of lines.
@lines = split('\n', $body);

# Reset the body variable.
$body = "";

# Add back to the body all the lines that do not start with the greater-than sign. 
foreach (@lines) {
        if (!($_ =~ m/^\>/)) {
                $body .= "$_\n";
        }
}

Plug-ins are executed according to the ~/emailgw/plugins/index file. This file contains a simple list of plug-ins that determines which plug-ins to execute and in what order. The gateway script will interpret every non-commented line (i.e., a line that does not begin with a hash sign) as a file name to open and run.

Perl Modules

The script depends on the following Perl modules:

• MIME::Entity
• MIME::Parser
• Mail::Send

These modules are not part of the standard Perl installation on CS server machines. These modules were therefore installed locally, under ~/emailgw/lib/. The gateway script uses the use lib directive to add this directory to the list of directories Perl searches for modules.

Attachments stored in eDocs

Any attachments are extracted and stored in the CS "eDocs" folder, available from the CS Web server. Within eDocs, there is a subfolder named "rtattach", which is then organized by STnumber. Web-based access is from this URL: https://www.cs.uwaterloo.ca/cscf/internal/edocs/rtattach UWdir credentials are required to view those files, which is further restricted to CSCF and MFCF personnel.

Samba access is also available via: \\smb-files.cs\edocs\rtattach