Tuesday, January 17, 2012

Peer Review

I realize that this isn't exactly timely, but who has time to put up posts in a timely manner? Not me. (I hope to have an untimely post on Coyne's hard determinism later in the week, too.) Anyways, via Leiter, I learn of this prediction of the demise of peer-reviewed models of academic publishing. (Or is it an attack? Hard to tell.) According to Aaron J. Barlow, associate professor of English at a CUNY college, "Blind peer review is dead. It just doesn’t know it yet."

Barlow argued that the traditional system of blind peer review -- in which submissions are sent off to reviewers, whose judgments then determine whether papers are accepted, with no direct communication with authors -- had serious problems with fairness. He said that the system rewards "conformity" and allows for considerable bias.

He described a recent experience in which he was recruited by "a prestigious venue" to review a paper that related in some ways to research he had done. Barlow's work wasn't mentioned anywhere in the piece. Barlow said he realized that the journal editor figured Barlow would be annoyed by the omission. And although he was, Barlow said he didn't feel assigning the piece to him was fair to the author. "It was a set-up. The editor didn't want a positive review, so the burden of rejection was passed on to someone the author would not know."

I agree that this sort of thing shouldn't happen, and that the editor was behaving inappropriately. (The editor's behavior is also extremely strange. Correct me if I'm wrong, but editors typically have a lot of discretion concerning submissions, and if the editor thought that the paper wasn't good enough for inclusion in the journal or didn't think it would be a topical or stylistic fit, or just didn't want to devote the pages to it, she has the freedom to reject the paper all by herself. It's not as though she is obligated to ensure that every submission is refereed.)

But I don't agree that this incident is an indictment of anonymous refereeing, or that even reveals any problem of any kind with blind peer-review procedures as such. The only connection between what this editor has done and the institution of blind review is that there has to be such a thing as blind review in order for an unscrupulous editor to employ it in bad faith.

The way I see it, there are at least three important reasons to retain double- (or triple-) anonymous (peer) refereeing in some form:

1. The presence of peer reviewed journals and presses acts as a gatekeeper. There's a huge amount of material out there, and it is helpful to have other people to do the work of figuring out which stuff is worth reading. There would be a lot more material out there if we went to an all self-publish model in which there were no gatekeepers.

Also, this is a great deal of help to us unknowns. The only reason why anybody has ever read my work is because I published it somewhere. (Or because I submitted it to them in a way that caused them to have a professional obligation to read it.) Somebody once cited my work, and I am absolutely positive that this would never have happened if it weren't for the journal I published it in.

2. The work is improved by the editorial process. I've had papers accepted "as is" only a couple of times. The rest of the time the papers were conditionally accepted if I could make certain changes. Sometimes these were suggested by the referees; others they were insisted upon by the editor operating independently of the referees. In each such case, the changes were worthwhile and made the paper much better.

To be fair, Barlow acknowledges this:

"I love the editorial process" when comments result in a piece becoming better, he said, and digital publishing allows this to happen easily. But traditional peer review simply delays publication and leaves decision-making "in the dark."

But Barlow is wrong about the darkness. The darkness is not bad. It is the key to everything:

3. The anonymity of blind review procedures are extremely important. They are the only way to protect authors from all sorts of cognitive biases. Any suggestion that there should be less anonymity than there is now is hopelessly naive. There should be a lot more anonymity, not less.

There's also a strange reference to "guessing about how to handle a 'revise and resubmit' letter." I've had some R&Rs. As a referee, I've handed out some R&Rs. A lot of my friends have had R&Rs. I've never heard of an R&R where you would have to guess about what to do. You get an R&R because the editor believes in the paper and thinks that, with a little work, it would be worth publishing. It doesn't make any sense for an editor to make the author guess about what she should do. And in every case I've ever heard of, the editor has done the sensible thing and just explained what to do.

And even if the editor engaged in the bizarre practice of making the author guess about what revisions she was asking for, it still wouldn't constitute an objection to the practice of anonymous peer review as such. It would constitute an objection to this kooky editor's oddball R&Rs.

Long live anonymous peer review.

--Mr. Zero


Anonymous said...

The work is improved by the editorial process. I've had papers accepted "as is" only a couple of times. The rest of the time the papers were conditionally accepted if I could make certain changes. Sometimes these were suggested by the referees; others they were insisted upon by the editor operating independently of the referees. In each such case, the changes were worthwhile and made the paper much better.

Is this really true? Here's a test, look up bunch of articles and see where there's a footnote thanking a referee. Do you think those points are really crucial to the respective papers?

In my experience, about 50% of the R&R requests are helpful, and about 50% are just esoteric hobbyhorsing. This is a genuine cost: answering a tangential objection could obscure the central idea of the paper or complicate its presentation. Hey, I'll still make the changes with the hope of publication.

Anonymous said...

Google is the reason that blind review is dead.

This year I have sent four articles out to journals. Thanks to google analytics I can almost certainly tell you what department my reviewers came from and with a little less certainty I can even guess who my reviewers were (depending on the size of the department this can be near certain if only one person works in my area).

I can do this because very often when my status changes on the editorial or submission manager website for the journal to "undergoing peer review" or "out for review" or what have you I end up receiving a couple of strange hits to my website.

Google analytics confirms that the search terms that landed many of them to my site are the respective titles of the papers I sent out.

It is possible of course that my reviewers sent in their reports and only afterwards looked me up. In some cases this is clearly not the case (the hits come early and long before the decisions).

So if at least half (let's be charitable) of my reviewers are checking me out AND if I can figure out who my reviewers are...in what sense is the process blind? In name only.

Mr. Zero said...

Is this really true?

It's true in my experience, which (obviously) may or may not be typical. And my experience with editors has not been uniformly positive. But it's been positive on the whole.

Here's a test, look up bunch of articles and see where there's a footnote thanking a referee. Do you think those points are really crucial to the respective papers?

I don't see how speculating on whether the points were crucial is a test. I think in oder for it to be a test, you'd have to actually find out somehow if the referees gave good advice.

Google is the reason that blind review is dead.

Don't put your unpublished papers online if you don't want people to find them on Google.

Anonymous said...


So it's my fault that reviewers are looking for my identity?

What if the papers were given at a conference and show up with my name associated with them?

How can I possibly be held responsible for that?

Mr. Zero said...

Hi anon 10:54/11:06,

I didn't say it was your fault, or that you are responsible. I didn't use any language that hinted at any kind of blame or responsibility. I gave you some advice about how to prevent people from finding you on Google.

Here's a strategy about how to handle conference papers (I didn't invent it; it was proposed in comments one of the many times this topic has arisen on this and similar blogs): use one title when you submit the paper to conferences, and a different (but related) one when you submit it for publication.

Again, I'm not saying you're responsible for your referees's bad behavior. But come on. You know that people are going to try to do look you up on Google. You can either take steps to prevent it, or insist that it's not your responsibility to do anything other than anonymize the paper for submission. But by now you must be aware that if don't take steps to prevent it, it will not be prevented. That's all I'm saying.

Anonymous said...

I don't think the major worry is just that they will find MY paper by googling. The worry as expressed in the Leiter conversation is that the reviewer learns that a famous philosopher has written the paper she is supposed to be reviewing, and that this damages the fairness of the blind peer review system

Ben said...

"Barlow's work wasn't mentioned anywhere in the piece. Barlow said he realized that the journal editor figured Barlow would be annoyed by the omission. And although he was, Barlow said he didn't feel assigning the piece to him was fair to the author."

There's also a conflict though if Barlow is cited surely? So editors shouldn't send papers to those who are cited nor those who work in the areas but aren't? Anyway, how does he know whether he was the editor's first choice or not - perhaps he was asked only after cited authors declined.

"There's also a strange reference to "guessing about how to handle a 'revise and resubmit' letter.""

My guess is that he's referring to the fact that some R&Rs are pretty much conditional acceptances, whereas others are effectively 'we'll not take this paper but we'd look at a different one in the area'. Sure the comments should make clear what revisions are requested, but there's still a big question as to what your chances are if you make them.

Mr. Zero said...

The worry as expressed in the Leiter conversation is that the reviewer learns that a famous philosopher has written the paper she is supposed to be reviewing, and that this damages the fairness of the blind peer review system

Yeah, I see the problem. I have several remarks:

1. I don't see this as a criticism of anonymous peer review as such. (Maybe it wasn't meant to be, but the IHE piece clearly is.) The criticism is not that the anonymity of the author or reviewer somehow causes some problem; it's that the anonymity is easy to break, and that causes problems. The criticism therefore presupposes the importance of anonymous review.

2. Perhaps this is a criticism of anonymous review procedures as they are currently typically conceived of and implemented, insofar as they fail to recognize the existence of the internet.

3. So conceived, the problem is not insurmountable. The problem stems from the fact that it is not now customary to "internet anonymize" journal submissions. This should become customary as soon as possible. Some language should be added to the standard submission instructions: "take your name off the front page; remove self-identifying references from the body of the paper; ensure that you cannot be easily identified as the author of the paper via the internet, including on your academic web-page, blog, online CV, or a website for a conference in which an ancestor of this paper might have been presented."

As I said before, this might require authors to change the title of the paper for the purposes of submitting it to the journal. Upon acceptance, editors should ask authors whether they want to to use the title the paper was submitted under, or a different one.

Editors should also explicitly instruct referees not to attempt to use Google to discover the author's identity.

If I'm wrong about any of this, I hope someone will let me know.

Anonymous said...

I referee quite often (about once a month, I guess this is often, since I don't submit one paper a month, I'm doing what I think is a fair share of the process) and have to exercise considerable self-restraint to not google the title. Especially if the paper is intriguing, or right in my field. But I manage - I've always been able to wait to google until after I submitted my report. And indeed, in about 90% of cases I find the author within 5 minutes.

Anonymous said...

Every piece I have published has been improved by revisions suggestions by reviewers (except the one piece that was accepted without request for revision).

I may not agree with all suggestions I've received (and in some cases I explained why I was ignoring certain revision suggestions), but I agree with the process.

There are crappy editors and petty readers, but that does not mean the system is flawed.

Anonymous said...

@ 2:20-

Do you not google if your verdict is R and R? Further, I've submitted at least a few somewhat negative reports (i.e. reports I would not deem to be R and R verdicts) that have still been accompanied by resubmissions. Googling after submitting my initial report could have conceivably unblinded the process.

Glaucon said...

Wayne Coyne is a hard determinist?!? Crimenentlies! My lips are on fire...

Anonymous said...

2:20 here: yes, when I give R&R or CA I wait until I have given a final (second-round) assessment. But in 80-90% of cases, my assessment is R. It has only once happened that an editor gave an author R&R decision after my R, because the other referee was very positive (I, however, thought the paper was unsalvageable). I then knew who the author was, but s/he never resubmitted.

Anonymous said...

Having worked as a graduate assistant on a minor, but still well distributed, "bind reviewed" journal (which shall remain nameless), I can attest that at least some so-called "blind reviewed" journals are not that at all. When we would receive a submission from a famous philosopher, the editor would usually accept it outright and use it as the leading article in an issue. Other decent enough submissions would sometimes be accepted outright based on whether the editor wished to have someone in his debt (e.g., you'd have an advantage if you worked in a fancy locale the editor might want to visit someday). Still others that were submitted from the editor's own former students might be accepted outright if they were at least decent, etc. We graduate assistants got to vet the remaining articles from "unknowns" in case something looked particularly brilliant. If so, these would finally be sent off for "blind peer review." It was a thoroughly politicized process and I was ashamed to be a part of it (but my graduate stipend required I do this work for the year). I shouldn't generalize based on my experience of one journal, but it was enough for me to question the reality of "blind review" even apart from the googling issues mentioned by others. Also, let me add that, as a former assistant of the editor, I could have gotten published easily in this "blind-reviewed" journal, but I was too disgusted by the process to submit anything!

Anonymous said...

I've never heard of an R&R where you would have to guess about what to do.

I was once expected by an editor to deal with comments by a reviewer who (1) spent most of the review complaining that a passing example mentioned as a very minor aid to the reader hadn't been expanded into a commentary on the recent history of the political question to which it made reference, and (2) then very briefly mentioned two themes s/he thought I should have included, but without giving any rationale, or any specific examples of work supposedly overlooked, or any detail at all in fact.

It's a pity that, beyond Andrew Cullison's journal surveys, discussions of the p.r. process tend by necessity to rely on anecdotes like this.

Diogenes said...

If anything is going to kill the "peer review" model it is the fact that editors and referees take too much damn time. As the pressures to publish increase across the board, resulting in even *more* submissions, this is only going to get worse.

It is a potential career-killer for junior faculty (of all sorts) and advanced graduate students to have to wait 9 months to a year for journals to get their shit together. We all know the actual act of refereeing takes no more than a couple of days, but editors have no leverage to use on their referees, and referees have no motive beyond their own conscience to act with alacrity. The system is beyond broken, and the only people getting consistently good results from it are the famous and established (yeah, yeah, cite your marginal cases).

There is no good reason not to move to an online, open-access model for scholarship. None. The point of all of this scribbling is to generate and disseminate ideas. So long as the gatekeepers sit on their hands and fail to their job, increasing the desperation of all, scholarship loses. This is a ridiculous system and everybody knows it.

And yes, I'm quite bitter about the fact that two different journals wasted close to three years of my time as they "reviewed" an article. What crap.

Anonymous said...

2:20 again. Long review times are horrible, I agree. David Chalmers once said that if you can't referee something within six weeks (I hope I don't misquote this, I can't find the actual thread back where he says this), you should decline the invitation to referee. I'm a bit more lenient on myself and try to get in my reports within 8 weeks - never more than 10 weeks. But honestly, apart from altruism, what incentives are there for a referee?
Listing of refereeing of 'services to the profession' in cvs count for nothing, you don't even get a formal acknowledgment in the journal (except for your name among many others in a yearly list of referees), you don't get paid, you don't even get a free copy of the paper that includes your comments and suggestions if it gets accepted. Given the scandalous amount of money academic presses like Springer, Elsevier and Wiley make, they should be able to pay referees something, if only a small fee. This would make referees more keen to provide reports quickly. As I am probably not the only non-tenured, non-tenure track postdoc who frequently referees, a small monetary incentive would make a difference. They do it for book publishing. Why not for journal publishing?

Mr. Zero said...

Wayne Coyne is a hard determinist?!?

It's not explicit, but I think there are elements of a fatalistic/deterministic outlook here.

Anonymous said...

Perhaps this has been conjectured before, but maybe paying referees a small fee would greatly decrease the time it takes referees to return submissions. Afterall, if I getting paid for it (if only a bit), I might think of my responsibilites quite differently. This change plus the aforementioned deadline approach might be effective.

Anonymous said...

This is slightly off topic (I think it might even be important enough for its own thread)... but... Can anyone illuminate me (us?) with respect to the Philosophical Quarterly's referee procedure? I've spoken to many, many colleagues about this and almost everyone agrees that the Philosophical Quarterly, more than anyone else, gives very poor, but yet really aggressive and nasty reviews. Is it because papers are often sent to grad-students at St. Andrews (who are not exactly well known for their humility)? The journal seems to have its act together with respect to giving timely decisions (probably down to an excellent editorial manager) but wouldn't it be great if they also sent out referee reports of the quality of, say, the Australasian Journal of Philosophy???

Anonymous said...

But honestly, apart from altruism, what incentives are there for a referee?

So refuse to referee. It is actually worse to accept to referee and then sit on it for a long time. I thought that's the kind of shit people do when they want to scoop other people.

Anonymous said...

Anon 12:57 - PQ seems to have a good reputation for helpful comments, at least according to Cullison's journal survey (and my experience).

Anonymous said...

I was asked to be a referee on a paper last spring, and it was the first time I'd ever been asked. I didn't know the conventions about how quickly you're expected to do anything, and I let the email sit for a bit while I wrapped up grading for the semester. Eventually the editor emailed me again and asked again, saying they needed to know soon if I could do it because it took them a long time to find a referee who could handle the two diverse areas that came together in this paper. (And they really were distinct. There really aren't very many people working on both.

Well, I said I could do it, but the editor didn't tell me how long I had, and then my hard drive failed, followed by my backup drive and my flashdrive failing five days later, within an hour of each other. I saved my dissertation because I'd emailed it to faculty, but most of my work for the past six months was gone (I had an old hard drive that had everything up to that point). Also, I was just scheduling my defense and finishing up my dissertation. Finally, the editor emailed me and said they usually take so many weeks (maybe six, but I don't remember), and it was getting to that point. I said I was approaching my defense and asked if I could do it once I submitted my dissertation (which I believe was only a few weeks away). The editor agreed, and I did it once my dissertation was submitted.

I gave a pretty full commentary on the paper that I hope somewhat compensated the author for the long
wait, but it wasn't exactly a situation where I could have just simply done it in six weeks. Those
other two issues were far more important for me than another week or two of waiting probably were for
the author.

I say this to point out that it isn't always as simple as a referee sitting on it for months and months
but often has complicating factors. Sometimes it takes a while to find a referee. Sometimes the referee has never done it before and doesn't know the conventions on how fast to do it. Sometimes things out of the referee's control slow them down. Sometimes things the referee places as a higher priority really should be a higher priority. In my case, I think all these things were true. So I wouldn't assume your referees are just being jerks. I wanted to give a careful report, and I think I succeeded in giving one, but I'm sure I wouldn't have if I'd tried to speed things up too much more.

Anonymous said...

This is slightly off-topic but seemed like the best thread here to raise the issue. I've just signed
a book contract for a book based on my dissertation, and I have articles out for review right now that come from my dissertation. These would be my first peer-reviewed articles, so it's important to me not to withdraw them, and the publisher is ok with them being out there for review right now, as long as it's not too many (and they're all right with the current number, given my current chapter plan). My worry is that they might take too long to be accepted, especially if I get an R&R on any of them. Here are my questions:

a. If any of them gets rejected, is it bad form (or even illegal) to submit them to a journal after I've signed the book contract? I know I can't submit anything after the book is actually published, but I'm not sure if I can before it's published but after the contract is signed.

b. What if one of these articles gets rejected, and I sent a different chapter in its place (to keep the number of articles the same as what I've now sent)?

c. If it looks like a journal is taking too long, is it proper (or even a good idea) to tell the editor that I've had a book contract accepted that includes that material, and I'd like a response soon so that I can ensure its publication before the book is published (which we were hoping might be out by the next APA Eastern)?

I'm interested in answers from people who have actual experience working with book publishers.

Polite Briton said...

Nasty aggressive reviews from the Phil Quarterly? I don't think the problem is that they give the papers to St. Andrews grad students, as this has been going on for at least 15 years - so before the St. Andrews PhD program had the reputation it now has. I'm inclined to suspect the problem might be that of using an overwhelmıngly *British* pool of referees. (But I don't really know why British referees would be like this.)

Anonymous said...

There must be *some* peculiarity to the way PQ chooses its reviewers; anyone in the know care to share what it is with us??? Or is it a masonic secret?

Anonymous said...

Barlow was played by an editor (allegedly). It was a set up: he was supposed to do the editor's job of rejecting for him. But Barlow saw through the scheme and refused to referee it!

Because he couldn't have, you know, read the thing and assessed it for quality. Oh no, such a thing would be beneath him or perhaps beyond him.

The problem with peer review is that there are idiot reviewers, like Barlow. Barlow should have stood up and said "peer review is broken! Proof: I'm asked to review things! Me! Dickhead that I am!" I would have been convinced.

Anonymous said...

Anon 6.47. You misremember Chalmer's remark. It wasn't 6 weeks; it was 72 hours. And it is reasonable. 8 weeks? Why? So you can spend 7 and a half weeks procrastinating? If you can't do it quickly, don't do it at all. And don't submit anything either: if you can't find the time to review a journal article, you certainly can't find them time time to write anything worthwhile.

Dale said...

'Realize' is a success verb. Did Barlow independently verify that this was the editor's intention, or does he simply have supreme confidence in his own intuitions? If his work had been referenced, would he then have assumed that the editor expected to accept the paper because he felt flattered? I'm frequently asked to referee papers on a figure on whom I've written that don't make reference to my work. I've never suspected that this was a nefarious plot on the editor's part.

By the way, I've refereed several times for the PQ and the instructions do not encourage nastiness. I used to referee frequently for the AJP, but they stopped asking me after I did write a fairly sharp report for what appeared to be undergraduate work. I assumed that they were simply too embarrassed to ask me again---I think it's the only time that I've actually complained to an editor about the fact that I was asked to review such obviously unpublishable work---but maybe they thought I was inappropriately rude. If so, they were wrong; it was entirely appropriate.

Anonymous said...

Wow, Dale. You seem really humble and sanguine about your abilities! I bow down to your amazing fucking intellect. I'm sure it's SUCH a great loss to the AJP that you're no longer reviewing for them.

DJ said...

This advice comes too late to help you, but it's worth mentioning for the benefit of later readers. If your hard drive fails, you replace it NOW. Not five days later. If you delay any amount of time, no matter how little, your backups will fail during that time. Your experience is a textbook illustration of this law of nature.