CS 854: Advanced Topics in Computer Systems:
Experimental Performance Evaluation

• Summaries must arrive in my mailbox before the deadline specified (usually this will be one day before, see each week for the specified time)
• Email each summary as a different attachment in the same SINGLE email. Use the Subject line "CS854 Summaries"
• Only pdf files are permitted. I'm going to use some pdf annotation software on my ipad to provide feedback.
• Use the naming convention yourgivenname-x-y.pdf where x is the number of the week (the first week with no readings was 0) and y is the paper number for that week. Numbers are provided beside each paper and week numbers are provided beside each week. Please be sure to double check them before summitting!
  For example if I was to submit the summary of the second paper listed in week one it would be named tim-1-2.pdf. If is was the ninth paper listed for week two it would be named tim-2-9.pdf.
  

Possible Readings

There will be two different types of summaries for different types of papers. Research papers will require research paper summaries. Other papers (e.g., survey papers) will require a survey paper summary. The format of these can be found by following the links below.

Paper Summaries

• Information and examples

Background / Helpful Readings

• Paper Reading Check List,
  Sugih Jamin, jamin@eecs.umich.edu.
• How to Read a Research Paper,
  by Spencer Rugaber
• Efficient Reading of Papers in Science and Technology
  Prepared by Michael J. Hanson, Updated by Dylan J. McNamee, January 6, 2000.
• How to Read a Paper
  S. Keshav, ACM SIGCOMM Computer Communication Review, Volume 37, Number 3, July 2007.

Readings / Course Schedule

• Week 0: Introduction
  Tuesday Sept 13, 2011
  • Course Overview
  • General Discussion
  
  
• Week 1: Assignment / Motivation
  Tuesday Sept 20, 2011 (Presentation Slides for assignment due Mon Sept 19, noon)
  • Presentations will be limited to 5 minutes.
  • Mini Assignment
  • Mini Assignment Presentations
  
  
• Week 2: Motivation
  Tuesday Sept 27, 2011 (Summaries for all 3 papers are due Sept 26 at noon)
  • Wrapping up talks from and discussion of assignment 1.
  • For each paper below do a Survey/Outline Summary as described in the information and examples for Survey/Overview papers.
    NOTE: These are fairly short papers, one is only 2 pages long.
  1. What is Experimental Computer Science?,
     Peter J. Denning,
     CACM 1980.
  2. Experimental Evaluation in Computer Science: A Quantitative Study,
     W.F. Tichy, P. Lukowicz, L. Prechelt and E.A. Heinz
     Journal of Systems Software, 1995.
  3. Should Computer Scientists Experiment More?
     Walter F. Tichy,
     IEEE Computer 1998.
  
  
• Week 3: Workloads and their Characterizations
  Tuesday Oct 4, 2011 (Summaries are due Oct 3 at noon)
  1. Workloads (Creation and Use),     (Survey Summary required)
     Alan Jay Smith,
     CACM, November, 2007.
  2. Web Workload Characterization: Ten Years Later     (Research Paper Summary required)
     Adepele Williams, Martin Arlitt, Carey Williamson, and Ken Barker,
     Web Information Systems Engineering and Internet Technologies, 2005.
  3. Towards Understanding Modern Web Traffic     (Research Paper Summary required)
     Sunghwan Ihm and Vivek Pai,
     Internet Measurement Conference, 2011.
  
  
  
• Week 4: Workload Generation
  Tuesday Oct 11, 2011 (Summaries are due Oct 11 by 9:30)
  1. Measuring the Capacity of a Web Server,     (Research Paper Summary required)
     Gaurav Banga and Peter Druschel,
     USENIX Symposium on Internet Technologies and Systems, 1997.
  2. Web workload generation challenges -- an empirical investigation     (Research Paper Summary required)
     Raoufehsadat Hashemian, Diwakar Krishnamurthy and Martin Arlitt,
     Software Practice and Experience, 2011.
  
  
  
• Week 5: Project Proposal Presentations
  Tuesday Oct 18, 2011 (PDFs of slides are due Monday, Oct 17 by 4:30 pm)
  Talks will be a maximum of 18 minutes long.
  The following people/groups will be presenting. The order will be randomized. Titles/topics are approximate.
  1. Ali - Obtaining 802.11 traces for use in rate selection algorithms
  2. Ray - An experiment to do with home energy consumption
  3. Ahmed - Evaluation algorithms for finding maximal pair in large strings
  4. Taha and Shehroz - Running Web Services in the Cloud
  
  
  
• Week 6: Tuesday Oct 25, 2011 (PDFs of slides are due Monday, Oct 24 by 4:30 pm)
  Talks will be a maximum of 12 minutes long with 12 minutes for questions = 24 minutes in total.
  The following people/groups will be presenting. The order will be randomized. Titles/topics are approximate.
  1. Noha and Michael - Web Server Scalability
  2. Alan - I'm not sure
  3. Aiman - Evaluating the performance of the Indri information retrieval systems
  4. Hamid - I'm not sure
  5. Tao - Performance Evaluation of TOR
  
  
  
• Week 7: Tuesday Nov 1, 2011 (Summaries are due Oct 31 by 9:30 am)
  1. A Nine Year Study of File System and Storage Benchmarking,     (Research Paper Summary required)
     A. Traeger, E. Zadok, N. Joukov and C.P. Wright
     ACM Transactions on Storage, 2008.

  Here is a follow up paper:
  Benchmarking File System Benchmarking: It *IS* Rocket Science,
  Vasily Tarasov, Saumitra Bhanage, Erez Zadok, and Margo Seltzer.
  HOTOS, 2011.
  
  

• Week 8: Tuesday Nov 8, 2011 (Summaries are due Nov 7 by noon)
  
  1. Week 8 Readings
  
  
  
• Week 9: Tuesday Nov 15, 2011 (Summaries Due Nov 14 noon)
  1. Producing Wrong DataWithout Doing Anything Obviously Wrong!     (Research Paper Summary required)
     Todd Mytkowicz, Amer Diwan Matthias Hauswirth and Peter F. Sweeney
     ASPLOS 2009
  2. Our Troubles with Linux and Why You Should Care.     (Research Paper Summary required)
     Ashif Harji, Peter Buhr and Tim Brecht
     APSYS 2011
  
  
  
• Week 10: Tuesday Nov 22, 2011
  1. Week 10 Readings
  
  
  
• Week 11: Tuesday Nov 29, 2011
  1. Week 11 Homework (Due Monday Nov 28, noon)
     Use filename = yourname-11-1.pdf and Subject: 854 Summaries
     
  2. Projects due BEFORE 9:30 am Tuesday Nov 29, 2011.
     Late projects will not be accepted.
     Use filename given previously and Subject: 854 Project
     
  
  
• Week 12: Program Committee Meeting Tuesday Dec 6, 2011 (Reviews Due Monday Dec 5, noon)
  • Information about Review Submissions (READ THIS FIRST)
  • Project papers
  • Project Reviews