CS 854: Advanced Topics in Computer Systems

Experimental Performance Evaluation

Fall 2020

Home Video Logistics Schedule
Projects Resources Integrity

Course Schedule and Readings

NOTE: Due to the switch from Mondays to Tuesdays we are currently short one meeting but we may leave things as they are due to the extra meeting added after the last official day of classes.

UNDER CONSTRUCTION: I'm adding more information and many of the links on this page will not work yet

If you have suggestions for papers you would like to read or topics you would like to cover please let me know.

Paper Summaries * NEW *

There may be two different types of summaries for different types of papers. Research papers will require research paper summaries. Other papers (e.g., survey papers) will require a survey paper summary. The format of these can be found by following the links below.

UNLESS OTHERWISE STATED SUMMARIES WILL BE DUE ON
Mondays at 11:59 am (i.e., midday / noon eastern time)
CLICK ON THIS LINK AND READ AND FOLLOW THE
INSTRUCTIONS CAREFULLY FOR HOW TO DO SUMMARIES
Information and examples

Recorded Class Videos

The recorded videos of the course meetings from each week can be accessed using the link below.

The required username and password have been posted to Piazza.
Class Videos

Week 1: Monday Sept. 14, 2020 (Topic: Introduction)

No readings
Introductions
Course overview and a bit of background
Information about the mini assignment (due next week)

Week 2: Tuesday Sept. 22, 2020 (Topic: Measuring WiFi Throughput)

Mini Assignment Due prior to Monday Sept 21 at noon.
Mini Assignment Instructions
Mini Assignment Presentations (limited to X minutes)
CHECKLIST
1. Read the assignment carefully.
2. Conduct the experiments.
3. Check back here to find out how long your have for your presentation.
4. Think about the message / story for your presentation.
5. Read the portion of the assignment "Some Tips for Giving a Presentation"
6. Create the presentation (be sure to use the points described in the assignment).
7. Generate a PDF of your presentation.
8. Submit the presentation using the naming scheme provided before the deadline.
9. Practice your presentation out loud ideally using an online video tool.

Week 3: Tuesday Sept. 29, 2020 (Topic: Variability)

Due Monday Sept 28 at 11:59 am (just before midday / noon)
NOTE: Papers 2a - 2d are each only about 1-2 pages (so there isn't as much reading as it may seem)

Some Statistics / Background (watch some Khan Academy Videos)
[NO SUMMARY REQUIRED]
How to read a paper
[NO SUMMARIES REQUIRED]
1. Paper Reading Check List,
  Sugih Jamin, jamin@eecs.umich.edu.
2. How to Read a Research Paper,
  by Spencer Rugaber
3. Efficient Reading of Papers in Science and Technology
  Prepared by Michael J. Hanson, Updated by Dylan J. McNamee, January 6, 2000.
4. How to Read a Paper
  S. Keshav, ACM SIGCOMM Computer Communication Review, Volume 37, Number 3, July 2007.
Conducting Repeatable Experiments and Fair Comparisons using 802.11n MIMO Network
[RESEARCH SUMMARY REQUIRED]
Ali Abedi, Andrew Heard and Tim Brecht
Operating System Review, 2015.

CHECKLIST (in order)
1. Create a PDF file that contains a simple bit of text like "Hello World"
2. When you get email from Crowdmark (I will set this up soon) with instructions for submitting your survey (it will likely be called an assignment), test that you can submit that file to Crowdmark as your solution. You will be replacing this submission later but you don't want to discover problems at the last minute.
3. Watch the Khan Academy Videos (for the topics you don't already know)
4. Read the information provided about Bootstrapping
5. Read papers 2a - 2d above
6. Read the instructions for producing a RESEARCH PAPER SUMMARY.
7. Read the example research paper summary
8. Read paper 3
9. Write a research paper summary for paper 3
10. Proof read your research paper summary
11. Edit your research paper summary
12. Proof read your research paper summary
13. Edit your research paper summary
14. Proof read your research paper summary
15. Edit your research paper summary
16. Generate a PDF of your research paper summary
17. Submit the summary before the deadline.

Week 4: Tuesday Oct. 6, 2020 (Topic: Background and Motivation) * NEW *

Due Monday Oct. 5 at 11:59 am (JUST BEFORE MIDDAY / NOON)

From Repeatability to Reproducibility and Corroboration
(Survey Paper Summary required -- skip the last question (i.e., just cover questions 1-3).
Dror Feitelson
Operating Systems Review, 2015.
Producing Wrong Data Without Doing Anything Obviously Wrong!
(Research Paper Summary required)
Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. Sweeney
ASPLOS, 2009.
Come up with a potential group and potential project. See more details in the CHECKLIST below.

CHECKLIST (in order)
1. Read the information above carefully regarding due dates and times.
2. Read the instructions for producing a SURVEY PAPER SUMMARY.
3. Read paper 1.
4. Write the summary for paper 1
5. Read paper 2.
6. Write the summary for paper 2.
7. Proof read your summaries multiple times, editing them and improving them each time.
8. Generate a PDF of both summaries.
9. Submit the summaries before the deadline.
10. Read the information on the course web page about what is required for the project.
  NOTE THAT THIS HAS BEEN UPDATED SO PLEASE READ IT AGAIN.
11. Reach out to people in the class about forming groups via Piazza.
12. Arrange to meet with potential group members via MS Teams, Zoom or whatever to discuss possible project ideas.
13. Choose a potential project idea.
14. Have only ONE member of the group click on the link below to send email containing the following information:
  - Full names of the group members (using the names the university knows about and uses :-)
  - The title of the project
  - A short one pargraph description of the idea behind the project. Note that this isn't the "Project Proposal" (that will come later). This will be used for approval before you actually do a project proposal.
  Click me to send email for Potential Group for CS854

Reading Week : Tuesday Oct. 13, 2020 (TBA)

No class (due to reading week)

Week 5: Tuesday Oct. 20, 2020 (Topic: Workloads and Benchmarks)

Due Monday Oct. 19 at 11:59 am eastern (JUST BEFORE MIDDAY / NOON)
Except for the Project Proposal (see below for details).
Read these papers in the order shown below
Note that the first two are relatively short and the 4th doesn't need to be read in excruciating detail.

Workloads (Creation and Use),
(No Summary required)
Alan Jay Smith,
CACM, November, 2007.
httperf -- tool for measuring web server performance
(No Summary required),
David Mosberger and Tai Jin ,
ACM SIGMETRICS Performance Evaluation Review (PER), Volume 26 Issue 3, Dec. 1998.
Methodologies for Generating HTTP Streaming Video Workloads to Evaluate Web Server Performance,
(Research Paper Summary required)
Jim Summers, Tim Brecht, Derek Eager, and Bernard Wong,
5th Annual International Systems and Storage Conference (SYSTOR), 2012.
Characterizing the Workload of a Netflix Streaming Video Server,
(No Summary required)
Jim Summers, Tim Brecht, Derek Eager and Alex Gutarin,
IEEE International Symposium on Workload Characterization (IISWC), 2016.
You are not required to do a really detailed reading of this paper and don't need to understand all of the results. Focus on what is being done, why, and how. Read the questions below first and ensure that you read the paper well enough to answer all of them.
Answer the following questions about this week's papers.
1. Why is it important to understand workloads?
2. Do you think httperf is a good tool or not and justify/explain your answer?
3. What is the difference between a workload generator, a workload and a benchmark?
4. Explain why you think you've been asked you to read (in order) papers 2, 3, and 4.
5. Describe *** exactly two *** (no more and no less) ideas for research that could be conducted as followup work to the "Characterizing the Workload of a Netflix Streaming Video Server" paper.
The answers to all questions must all fit on a single page (I expect most will be much shorter). Your solution must be submitted to Crowdmark as a PDF file.
Course Project Proposal: Submit a PDF slide presentation for your course project proposal. See the information on the course project page for details and please follow the instructions there.
**** NOTE: This is due Wed. Oct 21 11:59 pm eastern (just before midnight)

CHECKLIST (in order)
1. Remember to post your group members, project working title and short description to the Piazza thread.
  Piazza Group Project List
2. Read the information provided carefully regarding what you need to do and the due dates and times.
3. Start by reading the list of questions in point number 5 above to help guide some of your reading. Keep these questions in mind as you read the papers.
4. Read Paper 1.
5. Read Paper 2.
6. Read Paper 3.
7. Write the summary for Paper 3.
8. Proof read your summary multiple times, edit it and improve it each time.
9. Generate a PDF of your summary then proof read the PDF and edit if needed.
10. Submit your summary for Paper 3 using Crowdmark (this will be labeled Q3).
11. Read paper 4.
12. Create a document that answers the questions above about this week's readings.
13. Proofread the answers and produce a PDF for submission
14. Submit your answers to the questions using Crowdmark (this will be labelled Q5)
15. Submit your course project proposal.
  - Your file name must contain only the Waterloo email/userid (NOT student number) followed by .pdf This will be the group member that I will email feedback too (they will forward it to the other group members). For example if I'm the group member that email should be sent to, my file would be named brecht.pdf (because brecht is my Waterloo userid/email name).
    Following these instructions is important because I will have a script that looks at the filename and sends a copy of the marked up file to that Waterloo userid.
  - Send the file as an attachment by clicking on this link:
    Click me to email your Project Proposal

Week 6: Tuesday Oct. 27, 2020 (Topic: Conducting Evaluations and Common Mistakes)

Due Monday Oct. 26 at 9:00 am **** NOTE THE CHANGE IN TIME ****
I know that I said I would have two readings but instead of covering two research papers this week we are reading one paper and covering two other very short documents / videos.

The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations
(Survey Paper Summary required -- skip the last question (i.e., just cover questions 1-3).
Blackburn et al.,
Transactions on Programming Languages and Operating Systems, 2016.
Systems Benchmarking Crimes
(No survey required)
Gernot Heiser
Watch the this video until the 19:19 mark (feel free to watch more if you wish).
Common Mistakes and How to Avoid Them by Raj Jain.
(No survey required)
Here is a copy of the slides are used in the video (approximately the slides)
Common Mistakes and How to Avoid Them

The lecture and slides are based on (from) Raj Jain's textbook.
The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley- Interscience, New York, NY, April 1991,

CHECKLIST (in order)
1. Reread the post in Piazza "Writing issues / tips" (I've added some notes this week) and be sure to follow those tips when writing your surveys.
2. Read the information provided about what you need to do and the due dates and times carefully.
3. Read Paper 1.
4. Write the summary for Paper 1.
5. Proof read your summary multiple times, edit it and improve it each time.
6. Generate a PDF of your summary then proof read the PDF and edit if needed.
7. Submit your summary for Paper 1 using Crowdmark (this will be labeled Q1).
8. Read document 2 above.
9. Watch the video in 3 above and review the slides.

Week 7: Tuesday Nov. 3, 2020 (Topic: Dynamic Instrumentation)

Due Monday Nov. 2 at 9:00 am
**** NOTE THAT THERE WILL BE A TIME CHANGE ON NOV 1 ****
I suggest that you check the times carefully for the time zone that Waterloo, ON is in and your time zone and remember that you can always submit early.

Dynamic Program Instrumentation for Scalable Performance Tools,
(Research Paper Summary required)
Jeffrey K. Hollingsworth Barton P. Miller Jon Cargille,
HPCC 1994.
Dynamic Instrumentation of Production Systems,
(Research Paper Summary required)
Bryan M. Cantrill, Michael W. Shapiro and Adam H. Leventhal
USENIX ATC 2004.

OPTIONAL (BUT HIGHLY RECOMMENDED) READING/LISTENING:
A Wardrobe for the Emperor: Stitching Practical Bias into Systems Software Research
Brian Cantrill, Keynote Talk at USENIX ATC 2016.
Follow the link to the slides and/or the audio of the talk.
This talk presents some interesting views from one of the authors of Dtrace on USENIX, conferences, the review process, program commitee meetings and publishing.
I would recommend listening to the audio while following along with the slides (the slides are nteresting but the audio is much more illuminating).

CHECKLIST (in order)
1. Reread the post in Piazza "Writing issues / tips" and be sure to follow those tips when writing your surveys.
2. Read the information provided about what you need to do and the due dates and times carefully.
3. Read Paper 1.
4. Write the summary for Paper 1.
5. Proof read your summary multiple times, edit it and improve it each time.
6. Generate a PDF of your summary then proof read the PDF and edit if needed.
7. Submit your summary for Paper 1 using Crowdmark (this will be labeled Q1).
8. Read Paper 2 above.
9. Write the summary for Paper 2.
10. Proof read your summary multiple times, edit it and improve it each time.
11. Generate a PDF of your summary then proof read the PDF and edit if needed.
12. Submit your summary for Paper 2 using Crowdmark (this will be labeled Q2).

Week 8: Tuesday Nov. 10, 2020 (Topic: Miscellaneous)

NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK AND NEXT.
SEE THE CHECKLIST THIS WEEK AND NEXT FOR DETAILS.
YOU CAN CERTAINLY COMPLETE THINGS BEFORE THE DEADLINE (ESPECIALLY PICKING A TOOL).

Learning in situ: a randomized experiment in video streaming
(Answer the questions provided below -- instead of a paper summary)
Francis Y. Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein, NSDI 2020.
If you follow the link below you can also get access to the talk slides and a video of the talk.
Link to the NSDI page that contains talk slides and the video.
I don't know but I suspect that to answer the questions it would be best to be sure that you read the paper. Watching the video is not required but you may find it useful to help understand that paper. You might want to watch the talk first and then read the paper.
1. How would you rate the performance evaluations in this paper?
  Be sure to state what is good and bad about the evaluations and be clear which evaluations you are referring to.
2. How would you rate the paper in terms of the claims it is making? Describe if you think the claim(s) made are sound or not and explain your reasoning. Be sure to describe which claim(s) you are referring to.
Your answers must fit on one page (single sided). Use the same font size and margin restrictions as for summaries. Produce a PDF file to be uploaded via Crowdmark. Using point form is fine.
An introduction to Docker for reproducible research
(No Summary Required)
Carl Boettiger
Operating Systems Review, 2015.
Choose the tool you will describe for Week 9.
This is being done via Piazza.
- Working together as a group choose a tool that is used to help measure and/or understand performance and create a slide deck (presentation) describing that tool and what it can be used for. If possible include a small example of how it can be used and an example of the output.
- Examples of some possible tools include but are not limited to: gprof, Instruments, vmstat, iostat, oprofile, vtune, TAU, JMeter, iperf
- NVIDIA seems to have several GPU profiling tools.
- Here is a list of performance analysis tools
- This seems to refer to some tools for profiling python programs.

CHECKLIST (in order)
1. Choose a few tools your group would like to cover. Pick a few in case the one you want is taken.
2. Check that those choices have not already been picked by other groups.
3. Due by Wednesday Nov. 4. 11:59 pm (just before midnight).
  One group member post something to the Piazza thread I've created stating which tool your group would like to cover and the names of the people in your group (you will be using the same groups as for the course project).
4. Wait for me to approve the tool and its assignment to your group. I'll do this via the same Piazza thread. See Week 9 for further instructions and deadlines related to creating your and reading other groups' slide decks.
5. Read Paper 1.
6. Write the answers to the questions provided for Paper 1.
7. Proofread your answers, check the spelling, iterate, produce the PDF and then check it. If required, fix the PDF.
8. Due by Monday Nov. 9. 9:00 am.
  Submit the answers to the questions via Crowdmark.
9. Read paper 2.
10. Be prepared to discuss Papers 1 and 2 during the online class discussion.

Week 9: Tuesday Nov. 17, 2020 (Topic: Performance Tools)

NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK.
SEE THE CHECKLIST FOR DETAILS.

As a group, create a slide deck describing the tool you have chosen to present and upload them to the Piazza thread about the tools.
As individuals, pick and read two slide decks and be prepared to discuss and ask questions about those tools in class.

CHECKLIST (in order)
1. As a group, find different sources of information about the tool and learn about the tool.
2. As a group, try to use the tool to understand how you create a small simple example of how to use the tool.
3. As a group, create a slide deck (presentation) about the tool. It should take someone about 10 to 15 minutes to read through and understand the slide deck.
4. Due by Sunday Nov. 15, 11:59 pm (just before midnight).
  One member of the group submit the slide deck by posting to the Piazza thread (the same one used above). The slide deck must be a powerpoint file or a link to a Google slides presentation. If you use a link, after posting to Piazza, please have another group member check the link to be be certain that it can be access by anyone who has the link (without requiring a Google account or having to log in). Feel free to use speaker notes to help the reader, if you like.
5. Due Tuesday Nov. 17, 9:30.
  As individuals, choose and read through the slide decks from at least TWO groups.
6. Attend the online class discussion and be prepared to discuss the different tools, ask questions about tools other groups describe and answer questions about the tool your group covered.

Week 10: Tuesday Nov. 24, 2020 (Topic: Reproducibility in Machine Learning)

Due Monday Nov. 23 at 9:00 am

Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
(Answer the questions below about this week's paper)
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox, Hugo Larochelle
Answer the following questions about this week's paper.
1. In your own words briefly (in one paragraph) summarize the paper.
2. What if anything new did you learn from this paper?
3. Is there anything from this paper that you think you can apply to your course project? If so briefly describe what applies and why it is relevant to your project.
4. Provide any other points that you would like to make about the paper (optional).
OPTIONAL reading, if you are interested in finding out more about the "NIPS" experiments described on page 6 of this week's paper. This is the experiment conducted to examine the consistency of their reviewing process. You can find a starting point for reading below. Many people have interpreted that experiments as showing that getting your paper accepted to that conference (and possibly others) was/is fairly random.
The NIPS experiment
Eric Price, Dec 15, 2014,
If you know of or find a better source (starting point) please let me know.

CHECKLIST (in order)
1. Read the paper.
2. Create a PDF document that answers the questions for this paper. This document must be at most 1 page and follow the formatting guidelines used throughout the course.
3. Proofread and edit your answers to the questions.
4. Submit the answers to the questions via Crowdmark.

Week 11: Tuesday Dec. 1, 2020 (Topic: Variability in Clouds)

Due Monday Nov. 30 at 9:00 am

Is Big Data Performance Reproducible in Modern Cloud Networks?
(Write a Research Paper Summary)
Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, and Alexandru Iosup,
NSDI, 2020.
If you follow the link below you can also get access to the talk slides and a video of the talk.
Link to the NSDI page that contains talk slides and the video.
OPTIONAL reading, if you are interested this is our paper suggesting that RMIT should be used for cloud experiments.
Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments,
Ali Abedi and Tim Brecht, 8th ACM/SPEC International Conference on Performance Engineering (ICPE), 2017.

CHECKLIST (in order)
1. If you use a Zoom app to attend class, please update it to the latest release. If you use a browser, I think you should be fine.
2. Read paper 1.
3. Create a PDF version of your summary for paper 1
4. Proofread and edit your summary.
5. Submit your summary via Crowdmark.
6. If you like read paper 2.

Friday Dec. 11, 2020 (Topic: Project Paper Submission)

Friday Dec. 4 11:59 pm eastern (just before midnight): Abstract submission deadline
Friday Dec. 11 11:59 pm eastern (just before midnight): Paper submission deadline

Note that you can continue to submit new versions of your paper and abstract right up to the deadlines.
I suggest that you submit early and update your paper as you near the deadline.

Late submissions will not be accepted!!! Even a few minutes late. This simulates a real conference submission experience.
Submissions will be done using HotCRP
Use the link provided in the CHECKLIST below for submissions.
If you don't already have an account on you will first need to create and account and register your paper and all authors. If you already have an account, log in using your existing HotCRP account (a link is provided for this below).
All members of the group should create an account now as you will all eventually need one for reviewing. This will also allow any member of the group to submit and update the paper.
Be sure to create your account early and to submit early to leave time to fix problems if you do not pass the automated PDF checks, etc.

CHECKLIST (in order)
1. If you don't already have one create a hotcrp.com account.
  You can use this link.
2. Submit the abstract for your paper. You can change this later but it must be relatively complete and accurate representation of your project. https://uw-cs854-perf20.hotcrp.com
  Note that this is required to allow me to start planning reviewer assignments well in advance of the paper deadline.
3. Once you have submitted your abstract I believe that HotCRP will assign a number to your paper (e.g., 42).
4. Paper submissions will be anonymous, instead where the author list would appear, add your paper number (e.g., Paper #42).
5. Create a PDF of a draft of your paper that you wouldn't be embarassed to submit.
6. Submit a draft of your paper using this link https://uw-cs854-perf20.hotcrp.com
7. If there is a PDF checker on HotCRP for your submission, run it and fix any problems that might exist (e.g., typical issues are fonts and/or margins that are too small, etc.)
8. Continue working on your paper, improve the writing, proofread it and edit it.
9. Submit the final version of your paper using the same link https://uw-cs854-perf20.hotcrp.com

Wednesday Dec. 16, 2020 (Topic: Paper Reviews)

Due Wednesday Dec. 16 11:59 pm eastern (just before midnight)
Reviews will also be done via the conference web page (once the paper assignments have been made you will receive email explaining how to access the papers you need to review).
Reviews will be anonymous.
Background reading (not required but might be helpful)
Writing reviews for systems conferences, by Timothy Roscoe, 2007.

CHECKLIST (in order)
1. Log into hotcrp, read through and understand the review form you will be completing for each paper and download the papers you have been assigned.
  https://uw-cs854-perf20.hotcrp.com
2. Read your assigned papers.
3. Complete the reviews for each paper.
4. Check if your review is complete, sufficiently long, detailed and accurate (your reviews will be rated by other reviewers).
5. Submit your reviews. You will not be able to see other the other reviews for the papers you have been assigned until you have completed your review.
  https://uw-cs854-perf20.hotcrp.com

Friday Dec. 18, 2020 (Topic: Program Committee Meeting)

Friday Dec. 18 at 9:00 am eastern (NOTE THE SLIGHTLY DIFFERENT TIME!)
NOTE THIS IS AN EXTRA CLASS MEETING AND IT WOULD BE GREAT IF YOU CAN ATTTEND.
Past students have said that this meeting was very beneficial.

CHECKLIST (in order)
1. Log into hotcrp and read the reviews for all papers you've been assigned.
  https://uw-cs854-perf20.hotcrp.com
2. Note: you will not be able to view reviews for your paper submission until after some time after the program committee meeting has ended. You will recieve an email notification of the decision for your paper along with the reviews (or a link to the reviews).
3. Important: use the HotCRP system to rate each of the other reviews for each paper you have reviewed (i.e., rate other people's reviews).
4. Conduct an on-line discussion with other reviewers of the same paper.
5. Try to understand the different reviews and try to reach an agreement about whether or not each paper should be accepted.
6. Attend class where we will hold the conference program committee meeting.