CS846 Paper Review Form - Winter 2012

Reviewer: 
Rafael Olaechea
Paper Title:
Defining Domain-Specific Modeling Languages to Automate Product Derivation: Collected Experiences.

Author(s):
Tolvanen, Juha-Pekka
Kelly, Steven

1) Is the paper technically correct?
 [X] Yes
 [ ] Mostly (minor flaws, but mostly solid)
 [ ] No

2) Originality
 [ ] Very good (very novel, trailblazing work)
 [X] Good 
 [ ] Marginal (very incremental)
 [ ] Poor (little or nothing that is new)

3) Technical Depth
 [ ] Very good (comparable to best conference papers)
 [ ] Good (comparable to typical conference papers)
 [X] Marginal depth
 [ ] Little or no depth

4) Impact/Significance
 [ ] Very significant
 [] Significant
 [X ] Marginal significance.
 [ ] Little or no significance.

5) Presentation
 [ ] Very well written
 [X] Generally well written
 [ ] Readable
 [ ] Needs considerable work
 [ ] Unacceptably bad

6) Overall Rating
 [ ] Strong accept (award quality)
 [X ] Accept (high quality - would argue for acceptance)
 [ ] Weak Accept (borderline, but lean towards acceptance)
 [ ] Weak Reject (not sure why this paper was published)


7) Summary of the paper's main contribution and rationale
   for your recommendation. (1-2 paragraphs)


This paper shows how domain-specific languages are built on practice, using 20 case studies, showing four main ways of building them: based on domain's expert concepts, on the output  to generate, on the look and feel and on the variability space.  It categorizes each case study based on the approach(es) taken and then goes on to explain each approach. It describes that DSL's based on the output constructs to generate didn't raise the level of abstraction and created minor productivity improvements, however they did help in finding errors or design flaws earlier. It praises DSL's based on the look and feel combined with the variability space approach, saying this DSL languages raised the level of abstraction as they were all based on a Software Product Line (Software family) Platform and  the modeller could focus on the unique aspect of the product, however it recognized this type of DSM were the most difficult to create and they depended on a well defined variability space for the SPL and o modelling behavioural variability.

The main weakness of the paper was that it didn't present any of the case studies in detail, probably due to confidentiality issues. This meant we couldn't properly evaluate  his findings and had to take the author's synthesized  descriptions of the use cases.  However the categorization of the approaches to create DSL's and the findings that combining a "Look and Feel" approach with a "variability approach" tended to give the best results was very useful.


8) List 1-3 strengths of the paper.  (1-2 sentences each,
identified as S1, S2, S3.)


S1: It creates useful categories for DSL creation approaches based on a large number of case studies, helping highlight the way they are created on practice. 
S2: It pinpoints the main difficulty in creating DSL's in practice that increase productivity and abstraction are in modelling the behavioural aspects and the future variability of the software/ software product line.
S3: It shows how using the "look and feel" approach with the "variability" approach  to create DSL can help raise the level of abstraction and make use of the software product (family) platform to embed knowledge and code in the DSL, hence substantially raising productivity.

9) List 1-3 weaknesses of the paper (1-2 sentences each,
identified as W1, W2, W3.)

W1: None of the case studies are presented in detail.
W2: In the conclusions it claims in all cases there was an increase in productivity due to higher level of abstraction, which contradicts earlier statements that in cases using the output generation approach there were only minor productivity gains and the level of abstraction wasn't raised much. 
W3: All the research of the paper was based on cases were the DSL were created using MetaCase, hence there is a strong threat to validity to the study as to whether it could generalize, however they do recognized this weakness of their research.