Friday, February 5, 2010

"Sighted" Review Procedures

A little while ago, we had a brief discussion in comments about some apparent infelicities in blind review procedures. I'd like to subject this issue to a more complete discussion. Blind review procedures are deeply important; they are what separates us from the animals.*

Ideally, nobody who is in a position to evaluate a paper should be in a position to know the identity, institutional affiliation, career point, or other potential latent-bias-activating characteristics of the author. My question is, do any journals actually follow real, serious blind-review procedures? If so, which ones? Which journals almost follow them? Where do they deviate? Which journals don't follow them at all? And how can we get them all to be rigorously blind?

--Mr. Zero

* That and maybe some other things.

26 comments:

Anonymous said...

In the past Ethics had the sort of semi-blind process found at most journals- submissions were not blind to the editor and the editor made the first cut in deciding whether (and to whom) to send out papers for review. My impression is that most, or at least many, rejections came at this stage. It would be astounding if famous and well-respected people didn't tend to get the benefit of the doubt here. (This might even be justified, but it would be surprising if it didn't happen.) But, Ethics has now, under the new editor, Henry Richardson, moved to a more fully blind review process. I believe the editor might still see the author's name in some cases, but many fewer and not as a matter of course, as was the case before. This seems to me to be a clear improvement (even if it does make the submission process a bit more of a pain.)

Asstro said...

I think I have this right, but if memory serves from a panel at the Eastern APA (on which Henry Richardson was a panelist), Ethics engages in a "triple blind review" process. The papers are first blinded, sent to his and some associate editors' desks, which they then read. At this point, they'll conduct a desk review, sending about 75% of the papers back with a desk rejection.

If a paper makes it through the desk rejection phase, the editors then suggest some referees (still without knowledge of the author of the paper), and their managing editor checks to make sure that the referees are not the author or are not likely to reject the paper for stupid or personal reasons (as close compatriots or sworn enemies of the author).

The paper then goes out through the referee process, all to eventually land back with the editors. At this phase, the editors make a decision on the manuscript still without seeing the name of the author (allegedly). Most are either rejected or sent back as R&Rs.

But I think this actually raises other questions: if 75% of the papers get a desk rejection, how really blind is this, and how ultimately useful is this blind review process? Aren't the papers filtered by, effectively, a very small subset of the philosophical community?

Don't get me wrong. I can understand the need for desk rejections. Some papers are abysmal and should never be sent to a referee... or some just clearly need a lot of work. But to filter out huge numbers of submissions based on relatively private criteria, this seems problematic to me.

Incidentally, my "word verification" phrase was "suckno" How appropriate is that?

A1fifty8 said...

Good topic. This is a bit off-topic, but is worth mentioning here.

On several occasions, I've agreed to review papers that were then forwarded to me as Word documents. On a few occasions - and only after submitting my reviews - I checked the "properties" tab to see what came up. I wondered if these journals would protect my identity during the review process, should I submit papers to them. On two occasions, the properties tab revealed to me the names of the authors of the papers I had just reviewed.

Let me stress, I checked this only after submitting my reviews. I felt really weird about doing it, even then, but I was curious. And I was surprised by what I found.

Yes, I did contact the journals to make them aware of this. For myself, I wipe my info from Word documents I submit. Word has a security feature that erases private information when you save a document. But you have to do it individually for each document, at least on the version of Word I use now.

Anyway, journal editors and authors should be aware of this potential problem.

Anonymous said...

Like anon 11:31, I have checked to see who people are after finishing my review. I have tried to have fun with this area. You can change the data. Here are some text I have put in that line:

Fuck You Reviewer, jk.
Blinded For Review
Brian Leiter
Rutgers Philosophy
I'm a Lebowski
I see you too....

And some other silly stuff. Some papers make it, and some don't, but I have fun with it nontheless. I suggest everyone put some funny stuff in there.

Anonymous said...

With respect to Anon 3:09, especially since I love wit and all--in proper context. I've reviewed for many T-1 journals and checked those properties after the fact out of curiosity--had I encountered the nervy disrespect of "Fuck you Reviewer" for my curiosity about whether the author was clueless about digital information, I might have sent a followup to the editor to the effect that I stood by my call to publish/revise/reject--but that in any case the author is a complete asshole. So play with that if you will--but insult the reviewer at your peril, especially if the reviewer wishes to provide feedback to the editor about people who unintentionlly reveal themselves in that manner (as I have done).

Just so you know that your paper might come to thin-skinned me.

Jamie Dreier said...

Asstro,

The 75% is about right if you are including the managing editor's rejections and the associate editors' rejections, which are separate.

Only one associate editor gets any given paper for screening.

The managing editor does not actually cross check the referees against the author -- the associate editor does that after assigning a ref. So this means if I do assign a referee who turns out to be uncomfortably related to the author, I'll end up assigning another referee knowing who the author is (so the double blind is no longer in place). That's pretty rare, though.

It's true that 75% is high for unrefereed rejections. I'm not sure why you think this means the process isn't blind (you may have something there but I honestly don't see it). There's plainly a trade-off: sending every paper received to referees would keep the associate editors from wielding a lot of power, but it would enormously increase the load on referees and undoubtedly increase the already burdensome time from submission to decision. Keep in mind that Ethics is extremely selective, so one can often be very confident that even a good, worthwhile, publishable paper isn't going to make it into the journal.

For what it's worth, I believe that the biggest 'blindness' problem is that a referee (and less frequently an editor) can often recognize a paper that's been blinded -- maybe you heard it at a conference, or someone told you that this author had been working on this problem, or you just recognize the style or the distinctive view that's being pushed. I don't know what we can do about this, but it seems to me that it's a substantially larger threat to genuine blind refereeing than lack of an extra layer of blinding at many journals.

Anonymous said...

I had some concerns about this lately. It bugs me that some journals have editors that will know your identity and still do the refereeing. (Analysis doesn't do this with every paper, but some. I've heard rumors of other journals where the editor will referee a paper without being blind. Friends tell me that the editor has informed them of this.)

Here's a question. Suppose you think you're not getting a fair shake at some journal that isn't blindly reviewed. Is it wrong to use a pseudonym when submitting a paper? If it gets accepted, can they really complain that the author wasn't really Martin Lee Strumbucket?

Anonymous said...

10:04

But then once they find out who Strumbucket is (whether the real identity or even just the fact that it's a pen-name), you're back in the same boat. And if you use a different pen-name every time it'll just be really confusing.

Anonymous said...

Serious question: is it wrong to use as your pen name the same name as a famous philosopher? If I send a paper to Analysis using the pen name, "Anthony Brueckner", could anyone really complain (apart from, say, the real Anthony Brueckner)?

Anonymous said...

2:58

I think it would depend on why you want to do that. It's hard for me to imagine any motivation that would not entail treating some people only as means, so I would think it's wrong (and I'm not Kantian, I just agree with this principle). But I could just be overlooking a perfectly fine reason for doing it.

Anonymous said...

"I think it would depend on why you want to do that. It's hard for me to imagine any motivation that would not entail treating some people only as means, so I would think it's wrong (and I'm not Kantian, I just agree with this principle). But I could just be overlooking a perfectly fine reason for doing it."

If you think that your work won't get a serious read but think that other people will get a serious read when they submit work, it may well involve deception and if you think that we use the deceived as a mere means whenever we deceive, then there's a perfect duty you'd violate if you did it.

I don't think deception is always wrong (either all things considered or pro tanto). I don't think an editor can reasonably complain that an author used the name of someone else when submitting a paper if the editor's beliefs about the connection between author and name shouldn't play any role in the review process. The person who you are passing yourself off as might have reason to complain as you might make them look bad in the eyes of the referee if your paper is no good, but I guess you could ask for permission. If enough of us did this, it might make just a bit blinder a process that is supposed to be blind.

Anonymous said...

In defense of Feb 5, 10:04 PM:

Presumably, the question involves inventing a name (perhaps inventing a name of no really existing or dead/formerly existing person).

Presumably, an assumption/point of the question is to get around all kinds of differents versions of the "old boy network" and various cliques still quite active in analytic philosophy.

Depending on how blind the review procedures are, once the journal finds out your real name and that Strumbucket or whatever is a made-up name has been used could make a big difference if the journal finds this out after they have decided to accept the paper.

Many journals in the sciences are run in such a way that no one making any kind of decision about whether or not to accept a paper for publication knows the identity of the author until a final decision has been made and the journal has informed the author of that decision.

Many people in certain 'cliques' intentionally break/violate or simply do not use fully blind review procedures. I believe the spirit of the original post was to suggest a way to avoid or lessen chances of nepotism or prejudice on manuscript reviews.

Anonymous said...

Just a terminological point, but isn't it 'blind' review when the identity of the reviewer is concealed from the author (so they feel they can speak freely)? When the identity of the author is concealed from the reviewer, this is called 'double-blind' review.

Anonymous said...

obviously, there is no recognized standard definition for "blind" when it comes to journal manuscript review policies. Many of the best journals in the sciences are completely blind, so that the manuscript reviews are based on nothing but the content/merits of the paper. When the review process is not completely blind in this manner, a decision about whether to accept a manuscript for publication can, and very often is, based on factors other than the content/merits of the manuscript.

inquiring mind said...

3:28 writes:
Many people in certain 'cliques' intentionally break/violate or simply do not use fully blind review procedures. I believe the spirit of the original post was to suggest a way to avoid or lessen chances of nepotism or prejudice on manuscript reviews.

I'm just curious -- what evidence do we have for this? (Beyond "My papers don't get published and people's in certain cliques do"?)

Anonymous said...

I know people who do it. And I know people who know people who do it. It happens all the time. Of course, probably no one is going to admit doing it in writing (and certainly not on a public blog). It's hard not to spend much time in the field and not encounter such.

One could similarly ask what evidence do we have (in employment generally, not philosophy and academia particularly) that employers discrimate based on race or sex when making hiring or pay decisions? Of course, virtually everyone denies doing it, though it happens all the time. Many people are just sexist bigots, and some people unconsciously do it through all sorts of conditioned biases. Unfortunately, there are analogues in philosophy for manuscript submissions and probably for hiring decisions as well (though cases of the latter sort involving jobs are not as commonly known and openly discussed among cronies as cases of the former sort involving manuscripts).

See the second comment in the post on Troubled RSS Journal feeds for more claims about this topic.

Anonymous said...

"It's hard not to spend much time in the field and not encounter such."

I've been in the field for twenty years (seriously!). I've never knowingly encountered any subversion of blind review.

I've done some editing, distributed papers to referees. I've done lots of refereeing, and of course I've had my papers refereed. Not only have I never identified any subversion of blind refereeing, but I can't even see how it would be identified -- since, as you say, anyone guilty of it isn't going to talk about it openly.

"One could similarly ask what evidence do we have (in employment generally, not philosophy and academia particularly) that employers discrimate based on race or sex when making hiring or pay decisions?"

???
It's very important to find evidence of that, and it's not particularly difficult! How do you think employers are tried in court for violating anti-discrimination laws?

I think 8:44 is suspicious that you're talking out of your ass, and after this follow-up, I'm sharing the suspicion.

(My verification word is "bored".)

Anonymous said...

"Not only have I never identified any subversion of blind refereeing, but I can't even see how it would be identified -- since, as you say, anyone guilty of it isn't going to talk about it openly."

I can think of four pieces of evidence of the subversion of blind refereeing.

1. People saying in discussion threads that they've used the internet to find who authored a paper. I think their testimony provides some evidence.
2. I've caught people using the internet to trace my paper titles to my website multiple times. This has happened on the day that a paper went from some sort of editor assigned status to some sort of under review status at a journal that has online status updates. (I had sent paper to a journal. In a discussion thread last year, someone who worked at a department where someone had used google to discover that I had written that paper asked "How did you discover the identity of the person who searched your paper?" I explained it to him. This person identified themselves by name. It seemed for all the world that a person who specialized in my area who worked in the department of this person wanted to know how I knew that someone (him? a colleague with the same AOS?) could discover who was using the internet to subvert blind review.)
3. There are journals like Analysis that simply tell you that part of the process won't be blind. Some will be treated to blind review if they aren't rejected by an unblinded editor and aren't accepted by the unblinded editor.
4. There are editors who have written to tell me that they were going to be the referee of the paper. My guess is that they know whose paper they are editing since the email read, "Dear X, I'm going to be the referee for your paper, Y" If I know that this is a practice of the journal, I'm sure I'm not the only one. (Indeed, I am sure because I've had friends tell me that they've had similar experiences.) Can that be used for nefarious purposes? My guess is that it can.

Okay, so there's the evidence. I've caught referees with their hands in the cookie jar, we have referees in discussion threads admitting that they use what they can to subvert blind review, we have journals that have as a policy an unblinded process, we have editors who have taken it upon themselves to serve as unblinded referees (in some cases, they referee papers critical of their own work).

Anonymous said...

7:44

On your (1) and (2):
Yes, I agree that that kind of thing happens. I didn't realize that was the sort of thing you meant.
Why did you contrast "many journals in the sciences"? Is it impossible to google the author of a science paper? And what does this have to do with using a pseudonym? Wouldn't the antidote to being googled just be to keep the title and text of your paper off the web?
Finally, what does this have to do with cliques?

On your (3) and (4):
Obviously, some journals do not use blind review at all, and some use blind review only for papers that are sent to referees. Everybody knows this.

Anonymous said...

..."as you say, anyone guilty of it isn't going to talk about it openly..."

I think I typed something like: it is unlikely that people will admit doing this in print. Nonetheless, people talk about it all the time. What this has to do with cliques is that certain cliques review their cronies papers and recommend them for publication. Many of them do this without telling the editor that they are personal friends of the author and/or that they have discussed the paper with the author in private conversations for years. In addition, some editors of journals often acccept allegedly blinded papers by their former students (chapters of dissertations that they supervised, etc.), and/or jounral editors often send papers to referees who were former friendly classmates with the paper author. The blindness in such cases is a joke - it is fake, though many people play along. There are many other examples of similar activity/behavior/practice.

Anonymous said...

11:50,

7:44 here. I wasn't the person you were responding to initially. The remark about the sciences isn't mine. Sorry about any confusion.

Here's something that I tried once. I looked at the CVs of recent students that had worked with the editor of a very well known journal and was shocked to discover the percentage of publications these students had in this editor's journal. I did this because a friend who had worked with this editor had said that he'd send a paper to that journal because he couldn't get it published anywhere else. Success. He said he was pretty sure that the editor would know he wrote the paper and know to send it to a friendly referee who would know that he was refereeing the paper of a chum from graduate school. I don't want to piss in someone's soup, but the numbers I found looked damning. It looked like there were a lot of people who worked with a certain editor who had a lot of their publications in this journal. From what I could tell, students from this grad program were doing better landing papers in this journal than students from any other program and probably better than all the other programs combined. It's weird when someone has near half of their publications in a former prof's journal, right? It's weird when lots of former students of that editor have more than 20% of their publications in that editor's journal, right? This is mostly publicly available information, so we could do some studies and see if something stinks.

Anonymous said...

7 44 and 7 04, you seem to have the same sorts of things in mind as me, 5 49, no?

Anonymous said...

"7 44 and 7 04, you seem to have the same sorts of things in mind as me, 5 49, no?"

Yes, I think so.

Anonymous said...

7:04...I think I know what journal you might be referring to - can you give us a hint?

Anonymous said...

I am really skeptical about these complaints. The whole thing about "cliques" just sounds like sour grapes. I just don't believe that "some editors of journals often acccept allegedly blinded papers by their former students (chapters of dissertations that they supervised, etc.), and/or jounral editors often send papers to referees who were former friendly classmates with the paper author." I don't say this kind of thing never happens, I just don't believe anyone posting here knows that it "often" happens.

The story about a certain editor and his students' publications is harder to assess. It sounds like the kind of thing that does raise questions, doubts, eyebrows, etc. (I was at one point associated professionally with some philosophers who were associated professionally with the editor of a particular journal, and they and I agreed that it would be much better not to publish in that journal at all until they got tenure, just because it could be bad if the eyebrows raised belonged to a tenure letter writer. Publishing in that journal could be a tenure letter writer eyebrow raiser.)

It's hard to draw more of a conclusion than that from the abstract description. Of course, I can understand why you won't fill in the blanks -- you're right not to do so.

Anonymous said...

"I don't say this kind of thing never happens, I just don't believe anyone posting here knows that it "often" happens."

I wonder about "often" as well, but I suspect that it happens often enough to be a genuine concern. At any rate, I don't want to fill in the blanks and say which journal(s) I suspect. If people have their suspicions, they can do their own sleuthing. I did my little bit of research and didn't like what I saw, but I know it's not responsible to make any specific accusations.

"7:04...I think I know what journal you might be referring to - can you give us a hint?"

It's a philosophy journal that isn't Hypatia and isn't Phil Imprint. It's a little hint, we've narrowed the list to something like 183.