CS 854 Advanced Topics in Computer Systems: Experimental Performance Evaluation

Please follow the "Course Readings and Course Schedule" link to see the assignment information.

People trying to enroll in this course may notice that enrollment is closed. This is because the enrollment limit has been reached. There is a chance that people who have enrolled won't actually be taking the course for credit and I MAY be able to allow one or two extra students (but no promises). So if you want to take the course for credit please show up to the lectures and we'll try to sort things out.

UNDER CONSTRUCTION

Course Info

Instructor: Tim Brecht
email: brecht@cs.uwaterloo.ca (include CS854 in the Subject line !!!)
Day: Tuesdays
Time: 9:30-11:50
Location: DC 3313
Office Hour: DC 3508, Day and Time: TBA
URL : http://cs.uwaterloo.ca/~brecht/courses/854/

Grading (tentative)

15% Class Participation
25% Paper Summaries / Presentations
50% Class Project Paper (likely 6-8 pages)
10% Paper/project reviews.

Class Participation

Class Participation is an important component of this class. You are expected to have read the papers and be prepared to discuss them in class.

About the Course

Objectives : An in-depth study of issues, challenges, techniques, and pitfalls in conducting experimental performance evaluations in computer systems research. The course will focus on conducting scientifically sound experimental performance evaluations and drawing valid conclusions from the results. The goal is to help students to critically evaluate performance experiments in existing papers and to design and carry out experimental performance evaluations in their own work.
Course Meetings : Most likely weekly discussion of 2-3 papers but this format may vary. Some meetings will involve reading and critiquing the experimental performance evaluation performed in the papers read that week. We may perform some case studies, especially if students in the course are planning on conducting performance evaluations for papers or thesis work. This would involve a short presentation that describes a problem being studied and the methodology being used to evaluate the performance and the expected outcomes. The class will then evaluate and discuss the methodology.
Content : Some of the content may be dictated by the interests of the students.
MORE TO BE ADDED

Expected Background

These are not strictly required but many of the papers and discussion are likely to revolve around computer systems research so a good background in the following areas are highly recommended.

statistics
operating systems
networking
possibly some computer architecture
systems programming skills
good communication skills

Course Readings and Course Schedule

Hopefully some of this will be decided in conjunction with the class participants.
Possible Course Readings and Course Schedule

Some Resources Used or Talked about In Class

This might be a useful source / reference (and it's free).
Think Stats: Probability and Statistics for Programmers
by Allen B. Downey, published by O'Reilly Media.

V.S. Ramanchandran talk at Princeton. What Neurology Can Tell Us about Human Nature
V.S. Ramanchandran 2007 TED talk VS Ramachandran on your mind

Course Project and Paper

It will be possible to work on your own or in small groups of 2 or possibly 3.
The project paper will be due on the last day of classes.
After all projects are handed each student will read and provide a conference style review of some subset of the papers. This will be followed by a meeting that takes the form of a program committee meeting.
All projects and groups must be approved.
Information about what I'd like you to talk about for the project proposal.
WORKING DOCUMENT : The Final Project / Paper Logistics (THIS WILL CHANGE)

Academic Integrity

Note that students are not generally permitted to submit the same work for credit in multiple classes. For example, if a student has reviewed or presented one of the papers in another seminar class, he or she should avoid reviewing or presenting it again for this class.

The general university policy:

Academic Integrity: In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. [Check http://www.uwaterloo.ca/academicintegrity/ for more information.]
Grievance: A student who believes that a decision affecting some aspect of his/her university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70, Student Petitions and Grievances, Section 4, http://www.adm.uwaterloo.ca/infosec/Policies/policy70.htm. When in doubt please be certain to contact the department's administrative assistant who will provide further assistance.
Discipline: A student is expected to know what constitutes academic integrity [check http://www.uwaterloo.ca/academicintegrity/] to avoid committing an academic offence, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about "rules" for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate Associate Dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline, http://www.adm.uwaterloo.ca/infosec/Policies/policy71.htm. For typical penalties check Guidelines for the Assessment of Penalties, http://www.adm.uwaterloo.ca/infosec/guidelines/penaltyguidelines.htm.
Appeals: A decision made or penalty imposed under Policy 70 (Student Petitions and Grievances) (other than a petition) or Policy 71 (Student Discipline) may be appealed if there is a ground. A student who believes he/she has a ground for an appeal should refer to Policy 72 (Student Appeals) http://www.adm.uwaterloo.ca/infosec/Policies/policy72.htm.

Note for Students with Disabilities

The Office for Persons with Disabilities (OPD), located in Needles Hall, Room 1132, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with the OPD at the beginning of each academic term.