How I write reviews

The goal of peer review is to accept papers. The question is, given limited slots, which papers to accept? This is the question peer review seeks to answer.

Well-structured peer reviews make it easier for the program committee to rank and select papers for acceptance. I have a pretty specific template I try to stick to, originally inspired by Dan Cosley.

1. Summarize their argument

2-3 sentence paragraph.

Summarize the main point of the piece. Bonus points if you can highlight the intended contribution more crisply than the author. That's a great benefit to the author, and a great exercise for you.

2. Say what you liked about the paper

One paragraph.

Describe what was good about the paper. There's always a nugget of gold in every paper, even if it's just that the paper touches on some interesting subject matter. This is for the author's benefit: it highlights what's special, strong, or unique about their work and topic area.

3. Describe the major points that need improvement

One paragraph per point.

Get together the main weaknesses of the paper, and devote one paragraph to each. Don't go overboard here; there are never more than three big problems with any given paper. If there are, then there's probably only one problem: that the framing is poor (for example, that the work isn't well-motivated, or isn't well situated in prior work).

4. Describe minor issues

A bullet point list, one point per issue.

Describe minor qualms, gripes, typos, writing issues in a bulletpoitned list.

5. High-level recommendation

In a peer review context, give a high-level signal about whether you think the paper should be accepted or not.

A note on quantitative scores

While the majority of your review is your written piece, venues often ask for a quantitative score of the paper. Sometimes these will be Likert-style “strong accept” to “strong reject” type scales, sometimes something different.

Whatever you do, don’t give a neutral review. A truly agnostic review about acceptance or rejection is of no use to the AC. We need to move off the fence, not get on it. Tilt your quantitative score one way or the other.