~bjoern/advice/reviewing

You've been asked to review an academic paper for a conference or journal. Here are some thoughts on writing high-quality, useful reviews:

Purpose

First, understand the purpose of your review.

  1. You are a fact finder for your 1AC. You are one of several reviewers your associate chair (or 1AC) has assigned to this paper. Your job is to report back what the strengths and shortcomings of the paper are, given your expertise. Point out both positives and negatives. All papers have both. Based on all reviews, the 1AC and the program committee will take one of three actions: 1) reject the paper without discussion; 2) accept the paper with minimal discussion; 3) discuss the paper and possibly assign extra reviewers. If a paper is on the borderline in group 3, a detailed careful review is especially important to give the 1AC arguments during the discussion.

  2. Give authors feedback on how to improve their manuscript. No paper is perfect and your input helps authors improve their work in revision. This goal implies that a review is a piece of persuasive writing: you will need to phrase your criticism in a way that makes authors receptive to it. Be respectful of the hard work they have already invested. Do not use acerbic language. Argue about the work and not the authors, so the criticism is not taken personally. Point out positives as well as shortcomings. See Scott Berkun's How to give adn receive criticism (http://www.scottberkun.com/essays/35-how-to-give-and-receive-criticism/).

  3. Give your recommendation for acceptance or rejection. Avoid dithering: a rating of 3 (“borderline”) on a 5-point scale is often tempting, but least useful. Try to argue yourself off the fence, either for or against rejection. Match the impedance of the venue: if you are reviewing for a very selective, top tier conference, your the contribution for a high rating has to be greater than for a lower-ranked venue.

    Structure

    First, summarize the contribution of the paper in your own words, in one short paragraph. This paragraph should include both the technical contribution; the type of evaluation performed; and its results. State here if any important steps have not been performed, too (e.g.: “This paper introduces a new method for baking chocolate chip cookies based on heating dough with a laser cutter. The paper contributes an existence proof: cookies for three recipes were produced. However, the cookies were not tested for taste.”)

    Next, briefly give your recommendation. Don't bury the lede: Be kind to your 1AC and put the most important statements at the beginning of your review (right after your contribution summary). This includes your recommendation and a short, 1-2 sentence explanation of the main reasons for that recommendation. Organize your review with headings to facilitate skimming and random access. List review items in decreasing order of importance: typos and grammar checks go at the very end.

    Start by evaluating the big picture: How much of a contribution does this paper make to the literature? Is the result important, interesting, and sound? Does it go significantly beyond the state of the art? Magnitude of contribution should be the most important factor. Be biased in favor of important work with rough edges rather than unimportant, obvious work that is finely polished. There is a perceived bias of graduate students to be too negative on papers because of a myopic focus on details (i.e., missing the forest for the trees). Don't contribute to that perception.

    Review both the content and the form. Do the content first. The easy part is to criticize the words and images that already exist. Also spend time thinking about what is omitted in the paper. Are there important steps that are unclear or not described?

    Below are some topics you should address:

    Importance of problem

    Does the paper ask a clear research question? (It should.) Is the research question interesting and important? Is it timely? What benefits do we (the HCI community or society at large) gain from knowing the answer to the research question?

    Technical approach

    Are the techniques novel? Do you believe they will work? Do you have enough information to understand how they work?

    The first question is: has the proposed contribution already been made in prior work? To answer this question you may have to read up on the state of the art in the paper's area if you are unsure. One way to bootstrap your search is to identify the most closely related cited work in the paper; and then also look for other papers that cite those same papers using the ACM digital library. This process won't work if the related work section is insufficient to begin with, or if most publications in the area are very recent - you'll have to do your own background search.

    Sometimes, if the 1AC recruited you as an expert, you will have your own prior work that you feel the authors should cite. First, ask yourself whether your own paper really needs to be cited or whether you would like it to be cited. Only act in the first case. Write the suggestion in a way that allows you to remain anonymous - one strategy is to suggest at least two additional pieces of related work to discuss: one from someone else, one from you. Phrase both in the third person (e.g. “The paper's approach is similar to Hartmann [ref1], who introduced 3D printing of cookie recipies from XML files; and to Agrawala [ref2], who synthesized recipes using data mining.”) In any case, argue why you believe the work should be included. A paper that misses references but is fundamentally novel can be accepted. A paper that neglects to mention prior work which has already answered the same research question cannot.

    Evaluation:

    Think first about the purpose, the framing, and the overall results of the evaluation: Do you believe an evaluation is necessary to convince you of the merits of the paper? It's ok not to have an evaluation when none is needed. If an evaluation was performed — did it ask insightful questions? Can the experimental design answer the asked question? What did you learn from the results? Ask these fundamental questions first before arguing about proper techniques for statistical analysis. However, if you do find an error in the analysis method, do point it out.

    Formal aspects:

    Is the language clear and concise? It is easy to fall into jargon and abstraction in academic prose. Suggest where the writing can be more concrete. Are figures used appropriately? Are they legible? Are important figures missing?

    Rebuttals:

    If the venue has a formal rebuttal process, write out concrete questions you want to have answered in the rebuttal (e.g. “In rebuttal, the authors should clarify 1) whether their baking technique only works with brown sugar; and 2) how their technique differs from [ref1,ref2].”)

    Other people's thoughts: http://www.inf.ethz.ch/personal/troscoe/pubs/review-writing.pdf
    http://www.inf.ethz.ch/personal/troscoe/pubs/review-writing.pdf