CS 854: Advanced Topics in Computer Systems

Experimental Performance Evaluation

Fall 2022

 

 

 

 

 

Home Video Logistics Schedule
Projects Resources Integrity

 

 

 

Course Schedule and Readings

UNDER CONSTRUCTION: This page will get updated about once a week.
If you have suggestions for papers you would like to read or topics you would like to cover please let me know.

---

UPDATES:
Sat Nov 26 12:10:36 EST 2022: Added links and more information about completing the self/peer evaluations.
Sat Nov 19 14:15:26 EST 2022: Added Week 11 details, fixed HotCRP links, fixed links for course projects.
Fri Nov 11 15:40:09 EST 2022: Added Week 10 (and added some info and am working on remaining weeks)
Fri Oct 28 16:45:53 EDT 2022: Added Week 8 and 9
Fri Oct 21 15:48:47 EDT 2022: Added Week 7
Fri Oct 7 16:14:49 EDT 2022: Added Week 6
Fri Sep 30 17:10:07 EDT 2022: Added Week 5
Fri Sep 23 17:10:07 EDT 2022: Added Week 4
Fri Sep 16 16:23:41 EDT 2022: Added Week 3
Tue Sep 13 15:07:54 EDT 2022: Added information about maximum presentation length for assignment 0.
Sat Sep 10 09:56:11 EDT 2022: Added Background Readings Section

Paper Summaries

There may be two different types of summaries for different types of papers. Research papers will require research paper summaries. Other papers (e.g., survey papers) will require a survey paper summary. The format of these can be found by following the links below.

• UNLESS OTHERWISE STATED SUMMARIES WILL BE DUE ON
  Thursdays at 11:59 am (i.e., *** lunch / midday / noon eastern time ***)
  
  
• CLICK ON THIS LINK AND READ AND FOLLOW THE
  INSTRUCTIONS CAREFULLY FOR HOW TO DO SUMMARIES
  Information and examples

Background Readings (How to Read Papers)

• Some tips on reading papers (all very short).
• Required reading sometime before or during Week 3!!
• NO SUMMARIES REQUIRED

1. Paper Reading Check List,
   Sugih Jamin, jamin@eecs.umich.edu.
2. How to Read a Research Paper,
   by Spencer Rugaber
3. Efficient Reading of Papers in Science and Technology
   Prepared by Michael J. Hanson, Updated by Dylan J. McNamee, January 6, 2000.
4. How to Read a Paper
   S. Keshav, ACM SIGCOMM Computer Communication Review, Volume 37, Number 3, July 2007.

Week 1: Friday Sept. 9, 2022 (Topic: Introduction)

• No readings
• Introductions
• Course overview and a bit of background
• Information about the mini assignment (due next week)

Week 2: Friday Sept. 16, 2022 (Topic: Measuring WiFi Throughput)

• Mini Assignment Due prior to Thursday Sept 15 at noon.
• Mini Assignment Presentations (7 minutes maximum per presentation)
• Mini Assignment Instructions
• CHECKLIST
  1. Read the assignment carefully.
  2. Conduct the experiments.
  3. Check back here to find out how long your have for your presentation.
  4. Think about the message / story for your presentation.
  5. Read the portion of the assignment "Some Tips for Giving a Presentation"
  6. Create the presentation (be sure to use the points described in the assignment).
  7. Generate a PDF of your presentation.
  8. Submit the presentation using the naming scheme provided before the deadline.
  9. Practice your presentation while speaking out loud. Running through the talk silently in your head is often smoother, easier and faster than if you do it out loud.

Week 3: Friday Sept. 23, 2022 (Topic: Variability)

• Required summaries due Thursday Sept 22 at 11:59 am (just before midday / noon)

1. Read through the readings in the "Background Readings" section above
   [NO SUMMARY REQUIRED]
2. Part I: Statistics (some statistics background)
   NOTE: For now just go through Part I: Statistics on that page.
   [NO SUMMARY REQUIRED]
3. Conducting Repeatable Experiments and Fair Comparisons using 802.11n MIMO Network
   [RESEARCH SUMMARY REQUIRED]
   Ali Abedi, Andrew Heard and Tim Brecht
   Operating System Review, 2015.
   
4. Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments,
   [RESEARCH SUMMARY REQUIRED]
   Ali Abedi and Tim Brecht, 8th ACM/SPEC International Conference on Performance Engineering (ICPE), 2017.
   

• CHECKLIST (in order)
  1. Create a PDF file that contains a simple bit of text like "Hello World"
  2. When you get email from Crowdmark (I will set this up soon) with instructions for submitting your survey (it will likely be called an assignment), test that you can submit that file to Crowdmark as your solution. You will be replacing this submission later but you don't want to discover problems at the last minute. Seriously do this early.
  3. Watch Khan Academy Videos and do readings (for the topics you don't already know) that are listed in "Part I: Statistics"
  4. Read the information provided about Bootstrapping
  5. Read the Background Readings (about how to read research papers)
  6. Read the instructions for producing a RESEARCH PAPER SUMMARY (see section "Paper Summaries" above). NOTE: there are TWO different types of summaries. Be sure to always check and do the one being asked for.
  7. Read the example research paper summary
  8. Read the two research papers assigned
  9. Write a research paper summary for each paper
  10. Proof read your research paper summaries
  11. Edit your research paper summaries
  12. Proof read your research paper summaries
  13. Edit your research paper summaries
  14. Proof read your research paper summaries
  15. Edit your research paper summaries
  16. Generate a PDF of your research paper summaries
  17. Submit the summaries before the deadline.

Week 4: Friday Sept. 30, 2022 (Topic: Background and Motivation)

• Due Thurs Sept. 29 at 11:59 am (JUST BEFORE MIDDAY / NOON)

1. From Repeatability to Reproducibility and Corroboration
   (***SURVEY PAPER SUMMARY*** required -- skip the last question (i.e., just cover questions 1-3).
   Dror Feitelson
   Operating Systems Review, 2015.
2. Producing Wrong Data Without Doing Anything Obviously Wrong!
   (***RESEARCH PAPER SUMMARY*** required)
   Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. Sweeney
   ASPLOS, 2009.
3. Come up with a potential group and potential project. See more details in the CHECKLIST below.

• CHECKLIST (in order)
  1. Read the information above carefully regarding due dates and times.
  2. Read the instructions for producing a SURVEY PAPER SUMMARY.
  3. Read Paper 1.
  4. Write the survey paper summary for Paper 1
  5. Read Paper 2.
  6. Write the research paper summary for Paper 2.
  7. Proof read your summaries multiple times, editing them and improving them each time.
  8. Generate a PDF of both summaries.
  9. Submit the summaries before the deadline.
  10. Read the information on the course web page about what is required for the project.
      
  11. Reach out to people in the class about forming groups via Piazza.
  12. Arrange to meet with potential group members via MS Teams, Zoom or whatever to discuss possible project ideas.
  13. Choose a potential project idea.
  14. Have only ONE member of the group click on the link below to send email containing the following information:
      • Full names of the group members (using the names the university knows about and uses :-)
      • The title of the project
      • A short one pargraph description of the idea behind the project. Note that this isn't the "Project Proposal" (that will come later). This will be used for approval before you actually do a project proposal.
      Click me to send email for Potential Group for CS854

Week 5: Friday Oct. 7, 2022 (Topic: Workloads and Benchmarks)

• Due Thurs Oct. 6 at 11:59 am eastern (JUST BEFORE MIDDAY / NOON)
  
• Read these papers in the order shown below.
  Note that the first two are relatively short and the 4th doesn't need to be read in excruciating detail.

1. Workloads (Creation and Use),
   (No Summary required)
   Alan Jay Smith,
   CACM, November, 2007.
   
2. httperf -- tool for measuring web server performance
   (No Summary required),
   David Mosberger and Tai Jin ,
   ACM SIGMETRICS Performance Evaluation Review (PER), Volume 26 Issue 3, Dec. 1998.
   
3. Methodologies for Generating HTTP Streaming Video Workloads to Evaluate Web Server Performance,
   (Research Paper Summary required)
   Jim Summers, Tim Brecht, Derek Eager, and Bernard Wong,
   5th Annual International Systems and Storage Conference (SYSTOR), 2012.
   
4. Characterizing the Workload of a Netflix Streaming Video Server,
   (No Summary required)
   Jim Summers, Tim Brecht, Derek Eager and Alex Gutarin,
   IEEE International Symposium on Workload Characterization (IISWC), 2016.
   You are not required to do a really detailed reading of this paper and don't need to understand all of the results. Focus on what is being done, why, and how. Read the questions below first and ensure that you read the paper well enough to answer all of them.
5. Answer the following questions about this week's papers.
   1. Why is it important to understand workloads?
   2. Do you think httperf is a good tool or not and justify/explain your answer?
   3. What is the difference between a workload generator, a workload and a benchmark?
   4. Explain why you think you've been asked you to read (in order) papers 2, 3, and 4.
   5. Describe *** exactly two *** (no more and no less) ideas for research that could be conducted as followup work to the "Characterizing the Workload of a Netflix Streaming Video Server" paper.
   The answers to all questions must all fit on a single page (I expect most will be much shorter). Your solution must be submitted to Crowdmark as a PDF file.
6. Read Paper 4.
7. Create a document that answers the questions above about this week's readings.
8. Proofread the answers and produce a PDF for submission
9. Submit your answers to the questions using Crowdmark

Reading Week: Friday Oct. 14, 2022

• No class (due to reading week)

Week 6: Friday Oct. 21, 2022 (Topic: Conducting Evaluations and Common Mistakes)

• Summary due Thursday Oct. 20 at 11:59 am eastern (JUST BEFORE MIDDAY / NOON)
  Research/paper proposal due date/time is different (see below).

1. The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations
   (Survey Paper Summary required -- skip the last question (i.e., just cover questions 1-3).
   Blackburn et al.,
   Transactions on Programming Languages and Operating Systems, 2016.
2. Systems Benchmarking Crimes
   (No survey required)
   Gernot Heiser
3. Watch the this video until the 19:19 mark (feel free to watch more if you wish).
   Common Mistakes and How to Avoid Them by Raj Jain.
   (No survey required)
   Here is a copy of the slides are used in the video (approximately the slides)
   Common Mistakes and How to Avoid Them
   
   The lecture and slides are based on (from) Raj Jain's textbook.
   The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley- Interscience, New York, NY, April 1991,
4. Course Project Proposal: Submit a PDF slide presentation for your course project proposal. See the information on the course project page for details and please follow the instructions there.
   **** NOTE: This is due Friday Oct 21 12:59 pm eastern (just before class)

• CHECKLIST (in order)
  1. Read the information provided about what you need to do and the due dates and times carefully.
  2. Remember to post your group members, project working title and short description to the Piazza thread.
     Project and Group Member List
  3. Read Paper 1.
  4. Write the summary for Paper 1.
  5. Proof read your summary multiple times, edit it and improve it each time.
  6. Generate a PDF of your summary then proof read the PDF and edit if needed.
  7. Submit your summary for Paper 1 using Crowdmark (this will be labeled Q1).
  8. Read document 2 above.
  9. Watch the video in 3 above and review the slides.
  10. Submit your course project proposal.
      • Your file name must contain only a Waterloo email/userid (NOT student number) followed by .pdf
        This will be the group member that I will email feedback too (they will forward it to the other group members). For example if I'm the group member that email should be sent to, my file would be named brecht.pdf (because brecht is my Waterloo userid/email name).
        Following these instructions is important because I will have a script that looks at the filename and sends a copy of the marked up file to that Waterloo userid.
      • Send the file as an attachment by clicking on this link:
        Click me to email your Project Proposal

Week 7: Friday Oct. 28, 2022 (Topic: Miscellaneous)

• Due Thurs Oct 27 at 11:59 am (just before midday / noon)

1. Learning in situ: a randomized experiment in video streaming
   (Research paper summary is required and answer the questions below)
   Francis Y. Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein, NSDI 2020.
   If you follow the link below you can also get access to the talk slides and a video of the talk.
   Link to the NSDI page that contains talk slides and the video.
   BE SURE THAT YOU READ THE PAPER.
   Watching the video is not required but you may find it useful to help understand that paper.
2. This is not a separate paper but a set of questions to answer about the paper above. This will be completed and handed in using a separate document.
   
   1. How would you rate the performance evaluations in this paper?
      Be sure to state what is good and bad about the evaluations and be clear which evaluations you are referring to.
   2. How would you rate the paper in terms of the claims it is making? Describe if you think the claim(s) made are sound or not and explain your reasoning. Be sure to describe which claim(s) you are referring to.
   
   It is not required but it is fine if there is some overlap with some of the points in your summary when answering these questions.
   Your answers must fit on one page (single sided). Use the same font size and margin restrictions as for summaries. Produce a PDF file to be uploaded via Crowdmark. Using point form is fine.
   
3. An introduction to Docker for reproducible research
   (Survey paper summary is required)
   Carl Boettiger
   Operating Systems Review, 2015.
   

• CHECKLIST (in order)
  1. Read Paper 1.
  2. Write the summary for Paper 1.
  3. Write the answers to the questions provided for Paper 1.
  4. Proofread these documents, check the spelling, iterate, produce the PDF and then check it. If required, fix the PDF.
  5. Submit both via Crowdmark.
  6. Read Paper 3.
  7. Write the summary for Paper 3.
  8. Be prepared to discuss Papers 1 and 3 during the class discussion.

Week 8: Friday Nov. 4, 2022 (Topic: Reproducibility in Machine Learning)

• NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK.
  SEE THE CHECKLIST FOR DETAILS.
  YOU CAN CERTAINLY COMPLETE THINGS BEFORE THE DEADLINE (ESPECIALLY PICKING A TOOL).

1. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
   (No summary required. Instead, answer the questions below about this paper)
   Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox, Hugo Larochelle
   
   Answer the following questions about this paper.
   
   
   1. In your own words briefly (in one paragraph) summarize the paper.
   2. What if anything new did you learn from this paper?
   3. Is there anything from this paper that you think you can apply to your course project? If so briefly describe what applies and why it is relevant to your project.
   4. Provide any other points that you would like to make about the paper (optional).
2. The NIPS experiment
   Eric Price, Dec 15, 2014,
   (No summary required. Instead, answer the questions below about this posting)
   
   If you know of or find a better source (starting point) please let me know.
   This is the experiment conducted to examine the consistency of their reviewing process. You can find a starting point for reading below.
   Answer the following questions about this document.
   
   1. What is your understand and/or interpretation of the results of this experiment?
   2. Do you think that this will or will not influence how you think about paper acceptance and rejections? Justify and/or explain your answer.
3. Choose a tool that your group will describe for Week 9.
   See the information about Week 9 for more information about what will be required.
   This is being done via Piazza.
   • Working together as a group (using the same groups as for the course project) choose a tool that is used to help measure and/or understand performance and create a slide deck (presentation) describing that tool and what it can be used for. If possible include a small example of how it can be used and an example of the output.
   • Examples of some possible tools include but are not limited to: gprof, Instruments, vmstat, iostat, oprofile, vtune, TAU, JMeter, iperf
   • NVIDIA seems to have several GPU profiling tools.
   • Here is a list of performance analysis tools
   • This seems to refer to some tools for profiling python programs.

• CHECKLIST (in order)
  1. Read the first paper.
  2. Create a PDF document that answers the questions for Paper 1. This document must be at most 1 page and follow the formatting guidelines used throughout the course.
  3. Proofread and edit your answers to the questions.
  4. Due Thursday Nov. 3 at 11:59 am (just before noon)
     Submit the answers to the questions for Paper 1 via Crowdmark.
  5. Read the second document.
  6. Create a PDF document that answers the questions for Reading 2. This document must be at most 1 page and follow the formatting guidelines used throughout the course.
  7. Proofread and edit your answers to the questions.
  8. Due Thursday Nov. 3 at 11:59 am (just before noon)
     Submit the answers to the questions for Reading 2 via Crowdmark.
  9. Choose a few tools your group would like to cover. Pick a few in case the one you want is taken.
  10. Check that those choices have not already been picked by other groups.
  11. Due by Friday Nov. 4, 11:59 am (just before noon).
      One group member post something to the Piazza thread I've created stating which tool your group would like to cover and the names of the people in your group (you will be using the same groups as for the course project).
  12. Wait for me to approve the tool and its assignment to your group. I'll do this via the same Piazza thread. See Week 9 for further instructions and deadlines related to creating your and reading other groups' slide decks.

Week 9: Friday Nov. 11, 2022 (Topic: Performance Tools)

• NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK.
  SEE THE CHECKLIST FOR DETAILS.
  

1. As a group, create a slide deck describing the tool you have chosen to present and upload them to the Piazza thread about the tools.
2. As individuals, read through the slide decks and be prepared to discuss and ask questions about those tools in class.
3. As a group you will give a presentation about the tool you have chosen in class. Each presentation will be a maximum of 15 minutes and there will be up to 10 minutes for questions.

• CHECKLIST (in order)
  1. As a group, find different sources of information about the tool and learn about the tool.
  2. As a group, try to use the tool to understand how you create a small simple example of how to use the tool.
  3. As a group, create a slide deck (presentation) about the tool. It should take someone about 10 to 15 minutes to read through and roughly understand the slide deck.
  4. Due by Thurs Nov. 10, 11:59 am (just before noon).
     One member of the group must submit the slide deck by posting to the Piazza thread (the same one used above). The slide deck must be a PDF file.
  5. Due Friday Nov. 11, before class
     As individuals, read through the slide decks.
  6. Attend the class, give a presentation and be prepared to answer questions about the tool your group has chosen, and be prepared ask questions about and discuss the other tools.

Week 10: Friday Nov. 18, 2022 (Topic: Program Instrumentation)

• Summaries Due Thursday Nov. 17 at 11:59 am (just before noon)

1. Dynamic Instrumentation of Production Systems,
   (Research Paper Summary required)
   Bryan M. Cantrill, Michael W. Shapiro and Adam H. Leventhal
   USENIX ATC 2004.
   
   HIGHLY RECOMMENDED READING/LISTENING:
   Brian Cantrill, Keynote Talk at USENIX ATC 2016.
   A Wardrobe for the Emperor: Stitching Practical Bias into Systems Software Research
   Follow the link to the slides and/or the audio of the talk.
   This talk presents some interesting views from one of the authors of Dtrace on USENIX, conferences, the review process, program commitee meetings and publishing. I would recommend listening to the audio while following along with the slides (the slides are nteresting but the audio is much more illuminating).
   
   
2. SCALENE: Scripting-Language Aware Profiling for Python,
   (Research Paper Summary required)
   Emery D. Berger, 2020
   
   Link to a talk by Emery Berger that includes stuff about Scalene (Not required: but he's quite an entertaining speaker) "Python Performance Matters" by Emery Berger (Strange Loop Conference 2022)
   
   Scalene github repo

• CHECKLIST (in order)
  1. Read Paper 1.
  2. Write the summary for Paper 1.
  3. Proof read your summary multiple times, edit it and improve it each time.
  4. Generate a PDF of your summary then proof read the PDF and edit if needed.
  5. Submit your summary for Paper 1 using Crowdmark (this will be labeled Q1).
  6. Read Paper 2.
  7. Write the summary for Paper 2.
  8. Proof read your summary multiple times, edit it and improve it each time.
  9. Generate a PDF of your summary then proof read the PDF and edit if needed.
  10. Submit your summary for Paper 2 using Crowdmark (this will be labeled Q2).

Week 11: Friday Nov. 25, 2022 (Topic: Serverless Computing)

• Summaries Due Thursday Nov. 24 at 11:59 am (just before noon)

1. Peeking Behind the Curtains of Serverless Platforms
   (Write a Research Paper Summary)
   Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, Michael Swift
   USENIX ATC 2018.
2. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
   (Write a Research Paper Summary)
   Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini
   USENIX ATC 2020.

• CHECKLIST (in order)
  1. Read Paper 1.
  2. Create a PDF version of your summary for Paper 1
  3. Proofread and edit your summary.
  4. Submit your summary via Crowdmark.
  5. Read Paper 2.
  6. Create a PDF version of your summary for Paper 2
  7. Proofread and edit your summary.
  8. Submit your summary via Crowdmark.

Week 12: Friday Dec. 2, 2022 (Topic: Project Paper Submission and Wrapup)

NOTE THAT THERE ARE MULTIPLE DIFFERENT DUE DATES THIS WEEK AND NEXT.

• Wed Nov. 30 11:59 am eastern (just before noon): Abstract submission deadline
• Thur Dec. 1 11:59 am eastern (just before noon): Year End Questions due
• Friday Dec. 2, 11:59 pm eastern (just before midnight): Paper submission deadline

Please have a look at the form reviewers will be using to provide information about their view of your paper (this may be modified somewhat for the final form).
Review Form in Text Format
Note that this form is for completing a review offline. There is a more user friendly alternative available when you do your review online.

Note that you can continue to submit new versions of your paper and abstract right up to the deadlines (i.e., your abstract can change after your initial submission).

I suggest that you submit early and update your paper as you near the deadline.

• Late submissions will not be accepted!!! Even a few minutes late. This simulates a real conference submission experience.
• Submissions will be done using HotCRP
• Use the link provided in the CHECKLIST below for submissions.
• If you don't already have an account on you will first need to create and account and register your paper and all authors. If you already have an account, log in using your existing HotCRP account (a link is provided for this below).
• All members of the group should create an account now as you will all eventually need one for reviewing. This will also allow any member of the group to submit and update the paper.
• Be sure to create your account early and to submit early to leave time to fix problems if you do not pass the automated PDF checks, etc.

• CHECKLIST (in order)
  1. If you don't already have one create a hotcrp.com account.
     You can use this link.
  2. Submit the abstract for your paper. You can change this later but it must be relatively complete and accurate representation of your project. https://uwaterloo-cs854-perf22.hotcrp.com/
     Note that this is required to allow me to start planning reviewer assignments well in advance of the paper deadline.
  3. Once you have submitted your abstract I believe that HotCRP will assign a number to your paper (e.g., 42).
  4. Paper submissions will be anonymous, instead of where the author list would appear, add your paper number (e.g., Paper #42). DO NOT LIST THE NAMES OF THE AUTHORS ON YOUR PAPER.
  5. Complete and submit the Year End Questions (submit via Crowdmark) Year End Questions
  6. Create a PDF of a draft of your paper that you wouldn't be embarassed to submit (you can and are encouraged to resubmit updated versions of your paper)
  7. Submit a draft of your paper using this link https://uwaterloo-cs854-perf22.hotcrp.com/
  8. If there is a PDF checker on HotCRP for your submission, run it and fix any problems that might exist (e.g., typical issues are fonts and/or margins that are too small, etc.)
  9. Continue working on your paper, improve the writing, proofread it and edit it.
  10. Submit the final version of your paper using the same link https://uwaterloo-cs854-perf22.hotcrp.com/

Wed December 7, 2022 (Topic: Paper Reviews and Self/Peer Evaluations)

• Due: Wed December 7, 2022 at 11:59 am eastern (just before noon)
  Both the paper reviews and self/peer evaluations are due at the same time (plan accordingly)
  
  
• Paper reviews
  • Reviews will also be done via the conference web page (once the paper assignments have been made you will receive email explaining how to access the papers you need to review).
  • Reviews will be anonymous.
  • Required reading
    Writing reviews for systems conferences, by Timothy Roscoe, 2007.
• Self and peer evaluations

• CHECKLIST (in order)
  1. Log into hotcrp, read through and understand the review form you will be completing for each paper and download the papers you have been assigned.
     https://uwaterloo-cs854-perf22.hotcrp.com/
  2. Read your assigned papers.
  3. Complete the reviews for each paper.
  4. For each review please try include:
     • Good points about the paper (e.g., this might be why one should think about accepting the paper).
     • Bad points about the paper (e.g., this might be why one should think about rejecting the paper).
     • Reasons for your decision for Overall Merit and Experimental Methodology ratings.
  5. Check if your review is complete, sufficiently long, detailed and accurate. Your reviews will be rated by other reviewers and you will be graded on the quality of your reviews.
  6. Submit your reviews. You will not be able to see other the other reviews for the papers you have been assigned until you have completed your review.
     https://uwaterloo-cs854-perf22.hotcrp.com/
  7. Complete the self and peer evaluation (there is room for up to 3 group members, if you are in a group of 2 just leave the last column blank).
     
     • Use ONE of the following to complete the self and peer evaluation form.
     • Fillable PDF form
       Note that some PDF viewers appear to work for filling in forms but the values are not visible in other PDF viewers. See next step below to check that the values are visible in Crowdmark.
     • PDF form
       This is not a fillable form but could be used with a drawing tool. See next step below to check that the values are visible in Crowdmark.
     • Latex source
       You can download the latex source, fill in the table, produce a PDF file and submit the resulting PDF file (see the comments at the top of the file for details. This is probably the most fool proof method.
  8. Submit the PDF of the completed self and peer evaluation form using Crowdmark
  9. Log into Crowdmark and ensure that all of the values you have completed are visible. If the values are not visible, you probably need to use one of the other methods above.

Friday Dec. 9, 2022 (Topic: Program Committee Meeting)

• Friday Dec. 9 at 1:00 pm eastern
  
  
• NOTE THIS IS AN EXTRA CLASS MEETING AND IT WOULD BE GREAT IF YOU CAN ATTTEND.
  IT MAY TAKE SOME TIME BEFORE ALL OF THE LINKS BELOW WORK.
  Past students have said that this meeting was very beneficial.

After submitting your review for each paper and well in advance of the program meeting you should:

• CHECKLIST (in order)
  1. Log into hotcrp and read the reviews for all papers you've been assigned.
     https://uwaterloo-cs854-perf22.hotcrp.com/
     Remember that you will not be able to see other reviews for papers you are reviewing until after you have submitted your review.
  2. Note: you will not be able to view reviews for your paper submission until some time after the program committee meeting has ended. You will recieve an email notification of the decision for your paper along with the reviews (or a link to the reviews).
  3. IMPORTANT: USE THE HotCRP SYSTEM TO RATE EACH OF THE OTHER REVIEWS FOR EACH PAPER YOU HAVE REVIEWED (i.e., rate other people's reviews).
  4. Conduct an on-line discussion with other reviewers of the same paper.
  5. Try to understand the different reviews and try to reach an agreement about whether or not each paper should be accepted.
  6. Attend class where we will hold the conference program committee meeting.

<!-- ------------------------------------------------------------ --> <h3> Week 8: Tuesday Nov. 9, 2022 (Topic: Cloud Computing) </h3> <ul> <li> Due Monday Nov. 8 at 11:59 am </ul> <!-- <ol> <li> <a href=https://cs.uwaterloo.ca/~brecht/papers/icpe-rmit-2017.pdf> Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments, </a> <br> (Write a Research Paper Summary) <br> Ali Abedi and Tim Brecht, 8th ACM/SPEC International Conference on Performance Engineering (ICPE), 2017. <br> --> <li> <a href=https://cs.uwaterloo.ca/~brecht/courses/Perf-Eval-Shared/readings/f20/nsdi20-paper-uta.pdf> Is Big Data Performance Reproducible in Modern Cloud Networks? </a> <br> (Write a Research Paper Summary) <br> Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, and Alexandru Iosup, <br> NSDI, 2020. <br> Please read the paper. If you are interested can also get access to the talk slides and a video of the talk. <br> <a href=https://www.usenix.org/conference/nsdi20/presentation/uta> Link to the NSDI page that contains talk slides and the video. </a> </ol> <ul> <li> CHECKLIST (in order) <ol style="list-style-type:lower-alpha"> <li> Read Paper 1. <li> Create a PDF version of your summary for Paper 1 <li> Proofread and edit your summary. <li> Submit your summary via Crowdmark. <li> Read Paper 2. <li> Create a PDF version of your summary for Paper 2 <li> Proofread and edit your summary. <li> Submit your summary via Crowdmark. </ol> </ul> </ol>

 

 

 

Home Video Logistics Schedule
Projects Resources Integrity

 

 

 

Last modified: Sat Nov 26 12:42:17 EST 2022