How do researchers know whether they are doing “enough” or “too many” reviews? A measurable goal is to be review-neutral: to have demanded, through our submissions, as many reviews as we have produced as reviewers.

Reviewing is good

I like to review academic papers. It is a very rewarding activity in many different ways. One gets to serve the academic community, helping it function smoothly. One gets a chance at acquiring a much better understanding of someone else’s work than idle paper-skimming allows. One gets to send feedback to our colleagues and help them improve their work and its presentation — it is also an essential way in which we can participate to the formation of student researchers all over the world. Finally, doing reviews helped me develop the skill the judge someone else’s work and of forcing oneself to come up with a decisive opinion — it is surprisingly difficult and only comes with training.

Doing reviews is also fairly time-consuming. I noticed that the time I spend on each review is generally stable (excursions into previous or related work excluded): around one day and a half for conference reviews, and at least twice more for journal reviews — I’m sure other people have wildly different figures, but I would expect it to be a noticeable time commitment in any case. (Workshop reviews are much easier, at least for the formats I have seen of 2-page extended abstracts, I’d say one hour per review.)

How many reviews?

Because it is so time-consuming, deciding whether to say “yes” or “no” to invitations to review a new paper is not easy: in general I want to say “yes” (unless I can tell that I will not enjoy reading the paper at all), but it is not reasonable to say “yes” all the time, because I also need to spend time on other things. When should I say “no” because I have done “too many” reviews already?

We can count the number of reviews that we have done, and we can also estimate the number of reviews that we have demanded of others through our submissions. A natural goal for researchers is to produce at least as many reviews as they demand; if everyone reached this goal, the peer-review system would be at equilibrium without imposing too much of a workload on anyone.

To estimate the number of reviews a researcher demanded from their peers, you can sum, for each of their submissions to a peer-reviewed venue, the number of reviews that they received, divided by the total number of authors of the submissions.

\[ \sum_{p \in \mathtt{Submissions}} \frac{\mathtt{reviews}(p)}{\mathtt{authors}(p)} \]

Out of curiosity, I just measured this balance for myself: over my years doing research I have “demanded” 10 workshop reviews and 28.5 conference reviews, and “produced” 6 workshop reviews and 17 conference reviews. If you think that an article would interest me, you shouldn’t feel bad about asking me to review it, for now. (On the other hand, my balance this year is positive, so I wouldn’t feel to bad about refusing if I had to.)

Of course, a researcher’s balance is highly dependent on where they are in their academic career — maybe more so that on their personal choices. Students are supposed to submit articles, but are offered few opportunities for doing reviews. When they are invited to do reviews, it is often as sub-reviewer, one review at a time. More established researchers participate in program committees, where they have to do a fair amount of reviews at once — ten to twenty can be typical in Programming Languages conferences. This means that one naturally starts with a deficit of reviews, and that the opportunity to become balanced or positive only comes over the years.

(There is much more that could be said about the dynamics of the submission/review balance. I think the idea that a single person should be neutral should not be taken too seriously, because the dynamics are so complex. For example, some people stop doing reviews with a negative balance (students going to the industry for example), so long-time researchers necessarily have a very positive balance that may make short-time researchers balance considerations mostly irrelevant. Another thing is that there is no point doing more reviews than required by the submission flow, and that doing more reviews would build up more reviewing debt under this neutrality criterion — you can never have everyone positive.)

Quality

This is only a comment on the quantitative aspects of reviewing. Much more important is the qualitative part: are the reviews you receive and produce good reviews? (There is no objective definition of what a good review is; I like reviews that are constructive, help improve the work and its presentation, and catch mistakes.) For a given paper, one or a few very good reviews is more helpful than many bad reviews, so one should not compromise on the quality of one’s reviews in order to reach a quantitative goal.

Advice for students?

While proof-reading this post (thanks!), Ben asked some questions that may be of interest to others — mostly students, I suppose.

If I want to be review-neutral, but I have to accumulate a “review debt” before I can start reviewing, does this mean I should accept my first opportunity to review and every one that follows (until I’m neutral)?

The answer is of course “no”: one should never feel forced to accept reviews. On the other hand, I do think that it is worthwhile for PhD students to take advantage of the reviews they are offered, so “saying yes most of the time” sounds like a reasonable strategy to me — this is just a personal opinion. Some reasons:

• Reviewing is hard and takes training, I think it is good to start practicing early. Students are in a good situation to exercise their reviewing skills at a fairly calm peace (you won’t get many reviews anyway), and with more time than more senior people.

• Student reviews are often done as sub-reviewer: someone does a review, but also asks for your opinion and includes your sub-review in their review. It is a low-pressure way to do your first reviews, and the ability to exchange opinions with the other reviewer and discuss both reviews is really helpful. Students can also ask for feedback on their reviews to their advisor, which is also very helpful.

• Reviewing teaches a few useful things about writing papers as well — it’s always easier to recognize the flaws in others’ work.

On the other hand, I think you should not accept reviews at times when you cannot invest enough work in the review, or when doing so would be detrimental to you — whether you are on a deadline, or under too much pressure, or have conflicting commitments, etc. This is more important than anything about a submission/review balance.

Do you have any ideas for how young researchers / new researchers can reduce their “review footprint”? For example, is it possible to volunteer for reviews?

Yes, you can volunteer for reviews by telling the colleagues in your lab that you would be interested in doing reviews and that they should consider giving you some.

(With the increased use of double-blind submission processes, it is becoming more difficult to pass conference reviews to external researchers. This means that students are relatively unlikely to receive review offers from outside their close colleagues.)

Besides doing more reviews, the two other factors one could in theory play with are: submitting less papers, and having more co-authors. I think there is something to be said for the first one: one reason to not submit unfinished, buggy or topically-inappropriate articles is that it has a review cost. The second factor should not be considered, I think: “did this person contribute to the work?” should weight infinitely more for co-authorship decisions.

Note: Another thing you can ask for is reading reviews other people received. I think that reading reviews is also very helpful for research beginners — whether reviews of one’s own work or someone else’s. In particular, I wouldn’t know how to write reviews if I hadn’t had the opportunity to read reviews before that. If someone you are close to receives reviews, you should consider asking them whether you could have a look.

Is being a student volunteer at a conference equal to “one review”?

I think it is a distinct form of academic service. I don’t know how to measure the “conference organization cost” we impose to our academic colleagues. (If there are around 500 attendants to a typical Programming Languages conference, it means that for every 500 conferences you attend you should organize one all by yourself.)