CS848 Paper Review Form - Fall 2006

Paper Title: "A Comparative Evaluation of Transparent Scaling Techniques for Dynamic
              Content Servers"

Author(s): C. Amza, A. L. Cox, W. Zwaenepoel

1) Is the paper technically correct?
 [X] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [X] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [X] Good (comparable to typical conference papers)
 [ ] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [X] Significant
 [ ] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [X] Very well written
 [ ] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (very high quality)
 [X] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (marginal, willing to accept but wouldn't argue for it)
 [ ] Weak Reject (marginal, probably reject)
 [ ] Reject (would argue for rejection)

7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

  This paper presents a comparative evaluation of various combinations of transparent
scaling techniques in the context of clustered dynamic content web server systems.
The paper considers strategies for scheduling, load balancing, and query caching in
various sized clusters. The TCP-W benchmark is used to evaluate the performance of the
overall system configured with the different scaling techniques. A thorough experimental
evaluation is done to determine the best configuration in the context of the TCP-W
(e-commerce) workload. Conclusions are drawn about value of the choice of scheduling
algorithm, the lack of impact of the choice load balancing algorithm, as well as the
effects of varying load limits and the benefits of query caching. It is because of the
practicality of this paper, in evaluating the behaviour of current techniques in conjunction
with each other as a complete system, as well as the thoroughness of experimentation that I
rate this paper as "Accept".


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1: The paper presents a transparent solution for a complete dynamic content server
    system (no need to modify DB, web server, or app server). This adds to the
    practicality of deploying the proposed system.

S2: The paper presents a thorough evaluation and discussion of experimental results.

S3: It is (or claims to be) the first paper to evaluate combinations of these current
    methods for scalable dynamic content web server clusters. Previous research has only
    focused on specific details of each individual method.



9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1: The experiments do not address the effects of varying the number of schedulers 
    (though it is mentioned that 2 are used to ensure data availability).

W2: The experiments use "1 to 8 database server machines" but it is not always clear
    why the number was chosen (ex. Figure 7 shows the effect of load limits using 4
    databases, but later experiments use this result to configure the load limits of
    8 database clusters)


10) Detailed comments for authors.

The paper is very well written and is thorough in it's experimentation. It is interesting
to see the effects and performance of the various combinations of scheduling, load balancing,
load limiting, and query caching in a complete system. It would also be interesting to see the
effects of varying the number of schedulers, as systems (and cluster sizes) grow larger a
fixed number of schedulers likely won't scale. The paper doesn't address potential performance
issues between scheduler communication/synchronization and it is not clear if that would be an
issue.