Handling Security Alerts

We regularly receive alerts from IST's Security Operations Centre or notice unusual high bandwidth activity. Here we will make note of how to handle various alerts, including draft emails that can be sent to the user identified as being potentially involved. That person may or may not be aware of the activity as it may be that their machine has been infected. Always deal with such issues with great tact.

For IST's full handling procedures, see: IST Security Incident Response Procedures

Excessive Resource Use

Sometimes an individual running a program or making a CGI available causes noticeable problems on a CS server or the network. This may be caught in high bandwidth usage statistics (see below) or it may be discovered during investigation of a server issue.

Further investigation

Current process lists, lastcomm, and web server access_log can be useful in finding problem user/program combinations.

Sample Email, web CGI misuse

In the course of investigating problems with the __ web server, it came to our attention that a CGI installed in your personal web space is being used extensively by off-campus clients. The CGI in question is ___ [apparent function of CGI].

This CGI has been consuming excessive computing resources on the __ web server, contributing to problems __ (if any noted). It is also using a significant amount of the University's Internet bandwidth. In short, the program being hosted in your web space makes inappropriate use of resources, per UCIST guidelines http://www.adm.uwaterloo.ca/infocist/use.htm and UW Policy 71 http://www.ucalendar.uwaterloo.ca/0405/UW/policy_71.html.

Execute permissions have been removed from the CGI to prevent further abuse. (Suggest removing the program or running it on a non-university server). Web server logs are being monitored to ensure the program or any related web pages do not become active.

Followup, web CGI misuse

Set up a cron job to check web server logs for continued use of the offending program.

High Bandwidth

Various CSCF staff watch the bandwidth usage statistics
These statistics are also posted daily to the uw.network.stats newsgroup.

Further investigation

IST maintains the FlowViewer network flow reporting interface to provide detailed information about network traffic patterns. Dawn is familiar with the interface, as are some IST network and security staff. In particular, it can be useful to determine the times of day, network ports, and external IP addresses associated with high bandwidth usage.

In the case of a computer used by multiple individuals, login logs can be useful.

Sample report

They may send an email similar to:

somepc.cs sent out 25 GB of data on Saturday.  This seems unusual.

2009-06-06, 3056 GB external traffic: 1770 GB sent, 1286 GB rcvd

I=Internet,O=ORION        %day      MB   sentI  rcvdI  sentO  rcvdO
------------------------  ----  ------  ------  -----  -----  -----
...
somepc.cs(129.97.xxx.yyy  )        0.8   25766   25311    455      0      0

Sample email

We need to determine the owner of the machine and then send email, such as:

Subject:  High Network Bandwidth Usage

Dear ....,

We recently received a report that there has been an unusually high amount of network
traffic to or from your computer, as shown below:
...include relevant piece of the report...

If you are aware of the reasons for the activity and it is related to your research work
that should be no problem, just let us know and describe the nature of the activity.
If it not work-related, then please refer to the University guidelines on appropriate use
of network resources:
http://www.adm.uwaterloo.ca/infocist/use.htm

If you are not aware of any reason for such high activity, it may be that your machine is
infected in some way and should be checked for malware or unauthorized usage.  Please speak
to your Point of Contact, or reply to this email and we will have someone take a look at the
machine.

Thank you for your prompt attention to this issue,

..sender's name ...

or, for the student computing environment:

Dear ...,

Your account was logged into ... at ...date/time..., when it appears to have
...downloaded/uploaded... a large volume of data ...number from report....  This network usage
amounts to a significant portion of the University's external network bandwidth and may have
interfered with use of the Internet connection for teaching, research and other university
business.

Excessive use of the University's Internet connection is generally considered an inappropriate
use of resources, per UCIST guidelines
http://www.adm.uwaterloo.ca/infocist/use.htm
and UW Policy 71
http://www.ucalendar.uwaterloo.ca/0405/UW/policy_71.html.

If you are not aware of any reason for such high activity, it may be that your account is
infected in some way and should be checked for malware or unauthorized usage.

Please contact me to discuss your use of UW network resources.

... sender's contact information ...

Excessive Scanning

IST's Security Operations Centre runs software which looks for network traffic patterns indicative of compromised or attacking systems and send email to the DNS and abuse contacts for the system.

Sample report

Date: Sun, 7 Jun 2009 09:27:21 -0400
From: IST Security Operations Centre 
Subject: SECURITY(WARNING): Scanning(Excessive) at someserver.cs.uwaterloo.ca [129.97.7.XXX]
To: admin-contact@cs.uwaterloo.ca
Reply-to: abuse+reply@ist.uwaterloo.ca
Cc: abuse+reply@ist.uwaterloo.ca, dns-cs-admin@cscf.cs.uwaterloo.ca

Attacker: someserver.cs.uwaterloo.ca [129.97.7.XXX]
Attack: Network Scanning detected at cn-rtext router
Date: Sun, 07 Jun 2009 09:27:01 -0400

Automated monitoring processes have detected your system scanning for too many
network services -- well outside the norm. Network scanning has been detected
at levels which constitute good evidence of either gaming, a peer-to-peer
application (e.g., Skype and Bittorrent), a compromise or malicious intent.
Systems detected are routinely isolated from the campus network without
notice. See

     http://noc.uwaterloo.ca/cn/Stats/blocked

For our best advice on Skype see:

     http://ist.uwaterloo.ca/security/howto/2006-06-21/

You should determine why your system is scanning for so many network services
-- you may have a virus of some sort. An up to date version of Norton
Anti-Virus will help:

   http://ist.uwaterloo.ca/download/

University policies and the Resnet Statement on Misuse are clear that
network scanning of any sort will not be tolerated -- that includes scanning
programs to find open file shares. If you are running a scanner of that sort
you should stop now.

Resnet users should contact Resnet support staff:

   http://www.housing.uwaterloo.ca/resnet
   Phone: ResNet office at 519-888-4567 x33538 

If you need any help or assistance please let us know.


      IST Security Operations Centre
      
      Information Systems and Technology
      University of Waterloo, 200 University Ave W
      Waterloo, Ontario N2L 3G1 Canada


**** Flow Data Summary ****

1) Number of Flows observed from 129.97.7.XXX

2443

2) First few observations

0004 129.97.7.XXX     0001 65.109.171.24     06 a40e 50    9          729       
0004 129.97.7.XXX     0001 212.84.72.7       06 dd04 50    4          467       
0004 129.97.7.XXX     0001 81.169.145.69     06 8193 50    6          573       
0004 129.97.7.XXX     0001 115.28.148.229    06 a78c 50    14         920       
0003 129.97.7.XXX     0001 81.0.233.82       06 aa13 50    3          156       
0004 129.97.7.XXX     0001 205.211.183.4     06 850  c195  5          383       
0003 129.97.7.XXX     0001 216.197.111.238   06 829b 50    8          740       
0004 129.97.7.XXX     0001 217.76.142.21     06 ddc2 50    17         1142      
0004 129.97.7.XXX     0001 208.109.138.124   06 cad6 50    4          471       
0004 129.97.7.XXX     0001 82.98.135.43      06 d90d 50    5          520       

3) Last few observations

0004 129.97.7.XXX     0001 205.178.145.65    06 c47e 50    7          626       
0004 129.97.7.XXX     0001 216.230.225.3     06 d48c 50    9          729       
0003 129.97.7.XXX     0001 174.132.5.254     06 886a 50    9          729       
0004 129.97.7.XXX     0001 64.14.72.72       06 cc8b 50    6          574       
0003 129.97.7.XXX     0001 208.31.37.142     06 bef8 50    8          677       
0003 129.97.7.XXX     0028 216.251.43.98     06 e321 50    6          573       
0004 129.97.7.XXX     0001 72.47.229.3       06 c735 50    6          571       
0004 129.97.7.XXX     0001 67.210.107.83     06 8fdb 50    7          625       
0003 129.97.7.XXX     0028 64.26.174.74      06 a912 50    5          520       
0004 129.97.7.XXX     0001 206.171.93.39     06 b7d2 50    11         833       

4) Number of Peers Involved

2108

Sample email 1

We need to determine the owner of the machine and then send email, such as:

Subject:  Excessive scanning from your machine

Dear ....,

We recently received a report that there has been an unusually high amount of network
traffic to or from your computer, as shown below:
...include relevant piece of the report...

If you are aware of the reasons for the activity and it is related to your research work
that should be no problem, just let us know and describe the nature of the activity.
If it not work-related and/or involves copyrighted materials, then please refer to the 
University guidelines on appropriate use of network resources:
http://www.adm.uwaterloo.ca/infocist/use.htm

If you are not aware of any reason for such high activity, it may be that your machine is
infected in some way and should be checked for malware or unauthorized usage.  Please speak
to your Point of Contact, or reply to this email and we will have someone take a look at the
machine.

Please be aware that when there is evidence of a machine compromise, we will need to investigate
further to determine if there is any potentially sensitive information that may have been exposed.
Please do not wipe your machine, until instructed that it ok to proceed.

Thank you for your prompt attention to this issue,

..sender's name ...

Sample email 2

We need to determine the owner of the machine and then send email, such as:

Subject: Re: SECURITY(WARNING): Scanning(Excessive) at xxx.cs.uwaterloo.ca [129.97.xxx.xxx]
From: drallen@cs.uwaterloo.ca
Date: January 27, 2010 4:17:18 PM GMT-05:00

Hello, we've just received notice from IST security operations centre
that xxx.cs is performing scans and ssh brute-force attacks.  Unless
you know more about the situation, it looks like someone has
compromised the system.

Jason Testart wrote:
> I have disabled the port to xxx.cs since I've had several reports from
> on-campus of SSH brute-force attacks.
> jt

Since the system is likely compromised, I'd recommend a wipe/restore,
and changing your passwords on any accounts you have accessed via the
system.

Let me know if you would like assistance with next steps.

-Daniel

Following is the automated message we received earlier today:

Date: Wed, 27 Jan 2010 01:57:57 -0500
[...]
4) Number of Peers Involved

123 

Sample followup message, when the user has confirmed that they weren't doing scans themselves:

 Hi, I stopped by a few times and didn't catch you in.  The most
essential steps at this point are:

1) don't let this computer connect to any networks including wifi; it
may be able to "phone home" and allow access to the exploiter.  My
understanding is that its hardware port in the office is off and will
stay off until I tell IST the security problem is resolved.

2) if you used the laptop to connect to other services or servers, you
should change your passwords on all of them.  An assumption is that
anything you've done on the computer since it was exploited may have
been "sniffed" by the exploiter.  (It is possible that the exploiter
did not, but it would take some investigation to be sure).

3) Similarly: if you have private SSH keys that you could get to from
this laptop (either on the laptop, or on your CS accounts), you will
need to change those keys after you've changed your passwords.

4) Recovery of the system will require reinstalling the OS.  User data
may be compromised; we recommend (but don't have requirements about)
restoring your user data from known-good backups.

5) If student-related information is on the laptop, I'll need to know,
as part of campus Policy 8 on Computer Security Response Procedures;
the information will be shared with IST.

I'm around if you have questions/concerns; if I don't know the answers
to the questions I will figure out who does.

-Daniel 

Copyright Infringement

Sample Infringement Letter

Subject: Case ID 456555123 - Notice of Claimed Infringement
Date: Mon, 27 Jul 2009 11:47:38 -0400
From: MediaSentry 
To: Security Abuse Reports 

Monday, July 27, 2009


University of Waterloo
200 University Avenue West
Waterloo, ON  N2L 3G1  CA



RE:  Unauthorized Distribution of the Copyrighted Audio Work Entitled
       Pimsleur German - Pimsleur


Dear Security, Abuse Reports:

We are writing this letter on behalf of the Simon & Schuster Audio, a
division of Simon & Schuster, Inc., a a CBS Corporation.

We have received information that an individual has utilized the
below-referenced IP address at the noted date and time to offer
downloads of copyrighted television programs through a "peer-to-peer"
service, including such title(s) as:

     Pimsleur German - Pimsleur

The distribution of unauthorized copies of copyrighted works constitutes
copyright infringement under the Copyright Act, Title 17 United States
Code Section 106(3).  This conduct may also violate the laws of other
countries, international law, and/or treaty obligations.

Since you own this IP address (129.97.X.Y), we request that you
immediately do the following:

1) Remove or disable access to the individual who has engaged in the
conduct described above; and

2) Take appropriate action against the account holder under your Abuse
Policy/Terms of Service Agreement.

We also would request that you inform the individual who engaged in this
conduct that legitimate copies of Simon & Schuster Audio content are
available for purchase..

On behalf of Simon & Schuster, owner of the exclusive rights in the
copyrighted material at issue in this notice, we hereby state that we
have a good faith belief that use of the material in the manner
complained of is not authorized by Simon & Schuster, its respective
agents, or the law.

Also, we hereby state, under penalty of perjury, that the information in
this notification is accurate and that we are authorized to act on
behalf of the owner of the exclusive rights being infringed as set forth
in this notification.

Please direct any end user queries to the following:
CopyrightQs@mediasentry.com
 
Please include the Case ID 889380690, also noted above, in the subject
line of all future correspondence regarding this matter.

We appreciate your assistance and thank you for your cooperation in this
matter. Your prompt response is requested.

Respectfully,

A Kempe
Enforcement Coordinator
MediaSentry


------------------------------

INFRINGEMENT DETAIL
--------------------

Infringing Work: Pimsleur German - Pimsleur
First Found: 26 Jul 2009 10:07:35 EDT (GMT -0400)
Last Found: 26 Jul 2009 10:07:35 EDT (GMT -0400)
IP Address: 129.97.X.Y
IP Port: 58649
Protocol: BitTorrent
Torrent InfoHash: B8C527ABCD78ACF6D157454E293721AFDC5DF81C
Containing file(s):
Learn German.torrent (1,584,397,559 bytes)

Sample email to UW user

After doing some investigation to be as sure as possible regarding the likely user involved, send an email similar to the following:

Dear ,

We have received a complaint regarding a copyright infringement based on file(s) on your machine, see below.

If you are aware of the presence of this file, please make sure that it is removed and refer to the following regarding appropriate use of UW network and computing resources:
http://www.adm.uwaterloo.ca/infocist/use.htm

If you are not aware of how such files ended up on your machine, it may be that your machine is
infected in some way and should be checked for malware or unauthorized usage.  Please speak
to your Point of Contact, or reply to this email and we will have someone take a look at the
machine.

In either case, please respond to let us know that you have received this email and what actions you have taken.

Thank you for your prompt attention to this issue,

..sender's name ...

... Include full body of report here ...

Official response

Once the problem has been dealt with, send a report back to IST or the original complainant (to be determined) to let them know what was done.

Port pirating

Connections to lab computers or thin clients are sometimes unplugged so that people can plug in their own laptops. In order to make this work, the hijacker will need to allocate an IP address that will work on the port -- and typically (but not always) the IP address of the unplugged device is used.

Diagnosis

ONA can determine what MAC addresses are in use on a port. If the hijacker has ever used campus wireless with the hijacking device, the offending user can be tracked through ONA's authentication database.

How to notify the offending user

To be written/

SECURITY(WARNING): Windows Patches ...

These are sent out to warn when a system that had previously received patches from the UW WSUS server has not recently updated some critical patches. Note: in our dual-boot environment, this could be due to the fact that the user has been running Linux and not Windows.

Sample patch warning email

From: IST Security Operations Centre [mailto:soc@uwaterloo.ca] 
Sent: Sunday, May 01, 2011 3:31 AM
To: cs-rsg-bif@cs.uwaterloo.ca
Cc: abuse+reply@ist.uwaterloo.ca; dns-cs-admin@cscf.cs.uwaterloo.ca
Subject: SECURITY(WARNING): Windows Patches at xxx.cs.uwaterloo.ca [129.97.84.xxx]

The UW/IST Windows Software Update Service (WSUS) reports that your system is failing to install the following important Microsoft updates. 
These failures need to be investigated and resolved by human intervention. If you require assistance please contact your computing support representative.

Security Update for Windows XP (KB2481109)

If you need any help or assistance please let us know.


      IST Security Operations Centre
      
      Information Systems and Technology
      University of Waterloo, 200 University Ave W
      Waterloo, Ontario N2L 3G1 Canada

Sample email to user about Windows patching

... tbd