CS848 Paper Review Form - Fall 2006

Paper Title: Continuous resource monitoring for self-predicting DBMS

Author(s): Dushyanth Narayanan, Eno Thereska, and Anastassia Ailamaki. 

1) Is the paper technically correct?
 [x] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [x] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [x] Good (comparable to typical conference papers)
 [ ] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [x] Significant
 [ ] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [x] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (very high quality)
 [x] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (marginal, willing to accept but wouldn't argue for it)
 [ ] Weak Reject (marginal, probably reject)
 [ ] Reject (would argue for rejection)

7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)

The paper proposes a framework for quantitatively predicting performance
of a DBMS for a hardware resource upgrade, like memory, CPU, disks, and etc.
The proposed system can estimate new throughput, response time and where the 
bottleneck would be in a "what if" scenario of a particular hardware upgrade.
The main contribution of the paper is for using code traces from the database 
during an actual workload in order to create a model that can be used to predict 
troughput and response time for hypothetical resource configurations.

There is novelty in the paper's approach and its proof-of-concept prototype shows 
promise. The paper does a good job in illustrating its main ideas as well.

8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)

S1. Using code tracing during live execution of actual workload and the
    ability to create accurate hardware independent models to predict
    performance.
S2. The proposed architecture allows for analysis and modeling components 
    to be easily replaced.
S3. Trace data can be analyzed offline thus minimizing performance impact on
    the working system.

9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1. Only analyzed OLTP workloads, not even a prediction on DSS ones.
W2. Have to modify database engine code to add those traces, which 
    even when not in use could affect performance or cause increase code complexity.

10) Detailed comments for authors.

Buffer pools are not the only memory consumer to benefit from increased
system memory size. Such extra memory can be distributed to sort, locks, 
query parsing/compiling, different utilities, application agents, and etc. 
Such changes can significantly affect performance as well,
allow for a greater number of concurrent connections and even affect 
query plans suggested by the optimizer. Thus changes to more than one
memory consumer may have to be taken into account to predict the future
performance.

Larger disk space can affect database design produced by database advisor
tools like Database Engine Tuning Advisor from Microsoft or DB2 Design Advisor
from IBM. This can significantly alter workload performance as well and
effectively make another resource as the bottleneck.