Christopher’s Slightly Ornery Opinion on Writing SIGGRAPH Review Summaries (a.k.a. “Committee Comments for Authors”)

Having written, seen, and received my fair share of SIGGRAPH acceptance and rejection decisions, and served on a few SIGGRAPH Program Committees (PCs), I believe that we sometimes fail to give our review summaries the care and consideration that they deserve. With that in mind, I offer below my unsolicited personal opinions on how to write review SIGGRAPH summaries for authors. (Disclaimer: I definitely cannot and do not claim that I have always satisfied every item below! Rather, I consider these aspirations that I would like to live up to, and I hope you will agree.)

I’ll focus on the case of (conditionally) accepted papers, since one could plausibly argue they are most critical to get right, at least in the very short term. However, many of the sentiments below apply similarly to rejected papers.

• Know your audience – During the SIGGRAPH PC meeting, the Primary and/or Secondary reviewers are typically required to succinctly summarize the current state of the review decision (accept, reject, tabled, and why?) for a given paper, in text and/or verbal form. This terse and blunt review summary for the PC members should generally not be the same as the final review summary for the authors. The two contain related and overlapping information, but the audience and purpose are different. The former concisely narrates and provides insight on the decision-making process as it is happening, and is aimed at the PC members; the latter communicates the final decision to the authors, summarizes the factors influencing it, and describes in detail what the authors must do next.


• Be kind - Although you are probably a very tired person at this point in the lengthy SIGGRAPH PC process, the authors are people too and are therefore deserving of your consideration and respect. In fact, many authors are junior graduate students, whereas you are a leading expert in their field! As such, the statements that you provide, as the concluding evaluation of their work, can have a significant impact on those student authors. While they may still need to do further work to achieve final acceptance of their paper, those remaining requirements should be communicated kindly and constructively, rather than in an overly critical or disparaging way. Take the time and space needed to choose your words thoughtfully.


• Be precise - The review summary is a contract between the PC and the authors that outlines the conditional acceptance of the paper to SIGGRAPH: on the condition that the authors do X, Y and Z, the PC in return guarantees final acceptance. Therefore, the summary must describe for the authors what those conditions are, in a manner that is specific, clear, and verifiable, as much as possible. The review summary must also make a strong separation between the required changes and the recommended or suggested changes, i.e., provide two distinct, clearly labeled lists. The required changes should be only those changes that the reviewers collectively deemed strictly and scientifically necessary for publication, and are often those aspects upon which the acceptance decision hinged. This is not the opportunity to mold the paper into what you personally might have written if you had authored the paper, or to extract a pound of flesh in exchange for a grudging acceptance decision.
  To give some examples, “Do all the things mentioned in the rebuttal” is unacceptably broad and vague as a requirement. “Improve the paper in accordance with the reviews” can only be a recommendation, not a requirement, because it is essentially unverifiable; some of the many reviewer comments may also have been underspecified, incorrect, or mutually contradictory. Overly broad requirements leave the authors guessing as to what is actually vital, and gives the PC member carte blanche to pick and choose the actual hard requirements later – this is not appropriate or fair. “Evaluate your method on public dataset Y and provide a graph of runtime vs. problem size” or “Incorporate a brief discussion of the overlooked reference [XYZ]“ are some examples of more appropriate required changes. (See also: “SMART” goals.) On the other hand, it is fine for the recommended changes to be much more open-ended, since they are not intended to be strictly enforced.


• Be reasonable - While a motivated graduate student can accomplish an incredible amount, the revised paper has a tightly controlled page length and the revision process covers a finite time period (4-5 weeks or so). Do not require the addition of multiple pages worth of material to the main paper, unless you are also granting several additional pages of space. Especially do not place such requirements on conference papers that are already at the page limit, since additional pages are forbidden. If you require that a non-negligible amount of material be added, without providing additional space, you must specify what material should in turn be removed or migrated to a supplemental document; otherwise you have imposed an impossible requirement. (Also, don’t force authors into playing dubious games with layouts and font sizes and vspace and tiny unreadable figures/text, etc... which must then be undone later in the final revision/editing process.)
  Even more important than length, consider carefully the significant effort, time, and sometimes financial costs, that can be involved in performing additional prescribed experiments, incorporating and comparing against alternative public codes, implementing additional comparison techniques from scratch (usually frowned upon), fabricating new test objects, or other non-trivial revisions. Any or all of these can be perfectly valid in various circumstances, or they may be outright impossible in the time allotted. Regardless, the associated work often falls upon a single beleaguered graduate student or perhaps an outsider to the graphics community who is desperate to get final acceptance, and may have no idea whether the shepherd will offer any flexibility in their listed requirements. So be cognizant of the power dynamics involved and weigh each of your demands carefully.


• Be aware of the rules - Do not prescribe required changes that violate the current SIGGRAPH rules for the corresponding paper track. Read the rules carefully and obey them. As of this writing, for conference papers, changes are “limited to minor writing changes that are satisfiable within the current page limits and without requiring new experiments”. That is, no extra pages and no new experiments. For journal papers, new experiments can be and often are required, but final acceptance cannot hinge on the outcome of those experiments (at least, last time I served). Update: The rule regarding new experiments for conference papers has since changed (or rather been deleted): now reviewers can ask for whatever they want for either track, with the only stipulation being the conference track can't exceed the page limits... Another example is the distinction between conference and journal acceptance for dual-track submissions: the [current] rules indicate that the decision should be based solely on degree of evaluation/validation, whereas many reviewers/PC members tend to erroneously base it on [perceived] degree of novelty. If you violate such rules in your decision-making or specifying your requirements, the authors are put into a very awkward position: should they appeal to you? The PC chair to try to override you? Or simply tolerate the rule violation, inadvertent though it may be?


• Be responsible - While the Primary often writes the first draft of the review summary, they also generally run it (or at least the proposed required changes) by the Secondary, as well as other reviewers or PC members, to confirm that it is correct and acceptable. If you are one of those other individuals, don’t just assume the Primary has done proper due diligence and rubber-stamp it. Take a close look and suggest improvements, perhaps based on the suggestions in this list. It’s better to bother one of your fellow PC members with some quick changes to their Review Summary than to, for example, sentence a grad student to a hundred hours toiling away at an impossible, invalid, unnecessary, or contradictory requirement.


• Be congratulatory! - Lastly, SIGGRAPH paper acceptance is often one of the pinnacle moments of a computer graphics graduate student’s educational experience. It is the culmination of many months or years of study and diligent effort, perhaps coming after repeated and demoralizing rejections and revisions. With that in mind, take one extra sentence to directly congratulate the authors on their achievement. To be clear, I’m not saying you need to gush or be excessively effusive about the work. Simply opening with “Congratulations on your paper’s conditional acceptance to the conference/journal track of SIGGRAPH 20XX!” is perfectly adequate. Or drop the exclamation point if it’s not your style. By contrast, it is dispiriting to receive only a dull, perfunctory, and passive “The decision for this paper was conference acceptance” followed immediately by a sometimes harshly written list of reviewer criticisms and demands. Remember, we want these students to enthusiastically polish up their papers, prepare to give a great talk, and then come back to write more graphics papers in the future, rather than feel beaten down at their big moment of triumph. If it helps, consider how you would (I hope) deliver this exciting news verbally, face-to-face. This guidance holds even if your own initial review was on the negative side: the reviewers and PC members have collectively decided that this paper has value to the research community, and you are their representative. So, offer your congratulations as a small tidbit of explicit positive reinforcement to your fellow member of the computer graphics community.