Saturday, March 28, 2009

Some stuff about refereeing

There’s a nice discussion at Leiter about refereeing duties. Leiter has asked people to disclose the amount of refereeing they do, and to opine about how much refereeing one ought to do. A lot of people have contributed, and I think it’s a valuable discussion. I have a couple of thoughts, though.

1. Almost everyone who has responded claims to do what would have to be an above-average amount of refereeing. One person claims to referee 30 papers a year. Although I think that’s bullshit, a typical commenter claims to have refereed between 10 and 15 papers a year on average. I think that probably represents the high side. Nobody has said, “I never referee,” or, “I referee once a year because that’s the smallest amount that doesn’t leave me feeling like a total dick.”

2. Almost everyone who has responded claims to provide detailed feedback. I find that I almost never get feedback, and only very, very rarely is the feedback detailed. I have never gotten detailed feedback with a rejection—I’ve only received feedback with a rejection once, and while it was actually very helpful, it was not detailed. The detailed feedback accompanied a request to revise and resubmit (and was extremely, incredibly helpful—I’d like to know who the refs were so I could adequately thank them. Seriously.)

3. I have never been asked to referee a paper.

4. Franz Huber proposes a formal system of credits and debits that would guarantee smooth and efficient operation of the submission/refereeing system. I think the system would be disastrous—people who don’t get many requests to referee would be shut out, and this would much more likely to happen early on in one’s career, when you need to be free to send stuff out like crazy without being punished, but when no editor is going to think of you when thinking of potential refs. I think our current all-volunteer system, while flawed, is superior.

5. Matt S. asks a question I’d like to see answered: “how often does it happen that you get asked to referee a paper for one journal, reject it, then are asked by another journal to referee the same paper when it gets submitted there? Given how specialized most topics are, I would think this must happen with some regularity. If so, do you refuse to referee the paper, reject it automatically, reconsider depending on revisions or what the journal is...?”

--Mr. Zero


Anonymous said...

Also, another interesting question asked there by Ben Saunders:
"I have heard stories that suggest (some) journals will send papers they consider likely rejects (e.g. from grad students) to inexperienced reviewers, thinking it doesn't matter much, while papers from big name profs get sent to big name profs. Is this true of (some) journals?"

Almost-Doctor said...

I'm in my waning days of being a grad student. I've been asked to referee 5 or 6 papers in the last couple of years. Every time I get asked, I do the review, and I try and write about a page or two of comments within a month or two of getting the paper. I probably would not be able to do that for 15 papers a year, as it is reasonably time consuming.
More importantly, though, I think it's important to only agree to review papers that you can do in a month or so, and that you're willing to write comments on. It also makes sense to "pay in" to the system. I've reviewed more than I have submitted, but not quite double what I've submitted, which is
probably what it should be. But my excuse is that I don't get asked to review more often. Not that I particularly want to be.

Out of the 4 papers I've sent out for review, I have only gotten feedback on one, and it was an accepted paper. That's a bit irritating, but what irritates me more is having to wait a long time. My longest wait on a paper has been 12 months, and the median wait has probably been around 5 months. This just isn't fair to junior people. We need to demonstrate our qualifications for jobs, and build reputations. It's hard to do that if you can't even get a paper reviewed for a year. (This ignores the fact that it takes another year or two on top of that for your paper to make it to publication. The other half of this is that journals should post pdfs of papers shortly after they are accepted. Otherwise ideas take years to get disseminated.)

I like some version of a point system, to punish parasites on the system, but it can easily be fixed to support junior people. It would be nice if journals supported what in CS they call a "priority queue." It's the normal line, but certain factors can bump you to the top of the list for review. Junior papers should get assigned to people who typically turn papers around quickly. And if you're junior and have been offered to review something and you did it in a timely fashion, that should bump you above those that didn't.

CTS said...

To #5:

I've had this happen more than once. I always tell the editor I have reviewed a version of the same paper for another jounral and offer to take a look.

If the paper really has not been revised (Yes, this does happen), I send pretty much my original comments to the editor. If it is substantially different, I treat it as a 'new' piece so that I can review it with an open mind.

One of my worst reviewing experiences invovled a paper that I had suggested undergo some important changes but not full reversal of approach (I thought it had real promise). Another reviewer apparently trashed it, and the author follwoed that reader's advice to gut the argument. When I got the revised version, I thought it was terrible.

Lucky Asst Prof. said...

I have been asked to referee two papers this year. I refereed both of them and provided a full page of single spaced notes. I would probably be willing to referee about six a year with the same amount of rigor. I would consider doing more than that, but the quality would definately go down.

I have a new thread request:

I am fortunate enough to have my first monograph coming out next year with a moderately prestigious publisher. I believe that the book is good enough that it would be of interest to a broad variety of people in my AOS and might even make it into a few classrooms as a 'secondary-textbook' IF the book gets enough attention.

The publisher will do the typical promotional things that get hard-cover monographs like mine (with a price of $50-$65) into a couple hundred libraries. The publisher is not prestigious enough to put the book on everyone's radar.

What can I do to promote the book?


Jamie Dreier said...

Here’s some information based on my refereeing and editorial experience. (The editorial experience is distributing papers for refereeing for Nous, Ethics, and JESP.)

1. I don’t think the Leiter commenters are lying, but they’re certainly not typical – there is obviously a very powerful selection effect at work there. I referee approximately 8 journal papers per year (screening as an editor not included).

2. I think your experience is atypical. I usually get good feedback. (Early in my career I had a paper rejected by a very prominent journal with no feedback at all, so I never submitted there again and I won’t referee for them. Huh. Now that I've written it out, that sounds awfully petty!) I see lots of referee reports for Ethics and JESP and they are very good and detailed (in my opinion – I hope authors agree).
Ethics offers referees the opportunity to reveal their identity to authors. I think this is a very good idea.
Could it be that referees in ethics are better than refs in other subfields?

4. I agree, it’s a terrible idea. Besides the problem you mentioned, there is also the fact that it would set a low minimum and encourage senior philosophers to believe that their duty is done as long as they are breaking even in the credit system. This is a big mistake. Briefly: it’s true that the obligation to referee is an obligation of doing one’s fair share, but one’s fair share is not determined solely by the degree to which one uses the system (and esp. not by the degree to which one currently uses the system).

5. Often. I would say twice a year on average. I always decline.

Anonymous said...

Does anyone have an idea of how common it is for an editor to send a revised&resubmitted paper to a new set of referees rather than back to the one's recommending the R&R?

Anonymous said...

"If the paper really has not been revised (Yes, this does happen), I send pretty much my original comments to the editor. If it is substantially different, I treat it as a 'new' piece so that I can review it with an open mind."

I'm guilty of sending out a rejected piece without changing in light of comments and was unlucky enough to have the same referee get the paper the next time around. Predictably, the paper was rejected again. In my defense, the ref's criticism was obviously bad and my colleagues thought responding to it might actually do more harm than good. (Among other things, it is some indication that the paper has been rejected elsewhere, responding to the bad criticism disrupted the flow and could be distracting, etc...).

I've also found that there are some journals where I tend to get horrible comments as a rule where others have wonderful experiences as a rule. I suspect that I'm a victim of the editor sending my stuff to crappy reviewers. Maybe he thinks they are crappy papers, but I've landed some of these pieces in comparable journals or been given R&R's from better journals. Very annoying.

PA said...

Re anonymous @ 9:14:

Ditto. My policy is to not make changes to a paper for a journal that's not going to publish it. ( I did this early on in my career and ended up leaving out sections from the subsequently published version that I regretted). Refereeing is too much of a crap shoot -- on a number of occasions I have had papers rejected by very good journals only to have them subsequently accepted by other very good journals. I'd rather just throw the dice again.

Anonymous said...

A somewhat related question about blind review: when people submit papers for blind review, do they include a footnote thanking the people who have commented on the paper? If so, does that influence the review? That is, if I preface my paper with a note thanking Famous Philosophers 1, 2, and 3, does that influence how the paper is read? I've always removed such footnotes before submitting my papers, but someone over at Leiter seems to suggest that others leave this information in, and that this influences his review.

Anonymous said...


My current practice is not to include anything like that until my paper has been accepted for publication. Of course, I don't currently have any big-names to throw around. When the paper moves towards publication, there should be time to include such a note. With respect to whether such notes should count, I suppose that there could be some correlation between who is thanked and the final product, but when you have the paper in front of you, why not read the thing and make up your own fucking mind?

Mr. Zero said...


1. You should always take notes like that off. Leaving them on violates the spirit, of not the letter, of blind review.

2. It is obvious that the commenter does not suggest that others leave it in, or that this influences his reviews. The relevant claim (as Leiter points out about three comments down) is not a claim about how he reviews submissions. It is a claim that whether or not a paper is published is a less reliable guide to quality than the list of acknowledgments.

Popkin said...

Yes, it seems obvious that naming the people who have read the paper would undermine blind review. Here's what I'd like to know: is it standard practice to simply leave out acknowledgements and add them later if the paper is accepted, or is it standard to include a note for the acknowledgements, but replace the list of names with something like "[acknowledgements]"?

A more important question: will referees sometimes submit reports on papers when they know (or have a pretty good idea) who the author is? I would think that if someone is a specialist in the relevant area, they may have heard the paper delivered at a conference, or can simply recognize the author on the basis of the content of the arguments. Do referees generally feel they have an obligation not to write reports in such cases?

Anonymous said...

"is it standard practice to simply leave out acknowledgements and add them later if the paper is accepted, or is it standard to include a note for the acknowledgements, but replace the list of names with something like "[acknowledgements]"?"

The latter.