CS 741 Referee Report

Due: Tuesday, October 6, 2009 (in class)

Read the following paper:

Silvio Peroni and Fabio Vitali: Annotations with EARMARK for Arbitrary, Overlapping and Out-of-order Markup. DocEng '09: 9 pp.
and complete a 2-5 page review summarizing the paper's acceptability to the "CS 741 Selected Papers" series. You should address the following points (weighted as indicated). Use your own words and, except for the summary in the first point, do not merely rephrase ideas from the paper.

  1. Briefly summarize the paper. (1 mark)

    What is the intent of the paper? (Do not devote more than 1/2 page to this part of your review.)

  2. Describe the importance of the problem. (2 marks)

    Consider, for example, the following questions: Is it a new, significant, and relevant problem? Do the authors sufficiently motivate the problem? Did you learn anything? Does the problem fit the scope of the course?

  3. Describe the quality of the presentation. (2 marks)

    Consider, for example, the following questions: Do the authors cite the relevant literature? Do the authors clearly describe what was done and/or how it was studied? How well do the title, abstract, and summary convey the intent of the paper? Is the writing clear and concise? Do the authors provide the right level of detail? Are the figures informative? Are the mechanics (e.g., English usage) adequate?

  4. Evaluate the papers' contributions. (10 marks)

    1. Describe the quality of the research.
      • Enumerate three (3) of the paper's strongest points. For each one, indicate whether this is a major or minor strength. Explain your reasoning.
      • Enumerate three (3) of the paper's weakest points. For each one, indicate whether this is a major or minor weakness and elaborate on how the weakness could be corrected or eliminated.

      In addressing this part, consider, for example, the following aspects: Are the results described in the paper correct? Does the analysis use appropriate methods? Does the analysis cover all the important issues? Is the argument logical? How much has been implemented? Do the authors provide sufficient data and/or well-supported arguments?

    2. Describe two or three (2-3) of your own additional ideas for possible extensions to the work.

      You may wish to consider interesting short-term research or development that could be conducted within 6 months to explore the ideas further or help validate or repudiate the results. You could also consider long-term research or development that could be initiated to contribute more substantially to the ideas proposed in the paper.

      (Note that this part is especially important if you did not identify major weaknesses in the work. Explanations of how to overcome any of the identified weaknesses should not be repeated. If you choose to mention any of the extensions suggested in the paper, you must elaborate the ideas significantly beyond what is suggested.)

  5. Provide a numerical rating of the paper's acceptability. (0 marks)
    1 = excellent 2 = good 3 = fair 4 = neutral 5 = poor

Suggestion: You will likely find Keshav's essay entitled How to Read a Paper instructive.

Example reviews: Unfortunately, I do not have an example review that fits precisely the requirements of this assignment. However, some reviews of computing literature can be found elsewhere: