Wednesday, December 1, 2010

One Of Us Has Missed The Point

In a recent thread at Leiter, in which Thom Brooks posed some questions about referee guidelines, an anonymous graduate student comments:

All the comments so far seem to forget that blind review goes in two directions. A senior faculty member in my department likes to point out that she very often knows who wrote the articles she reviews, but the important thing is that the authors don't know she's the one reviewing them -- which means she can be fully honest in her assessments of their work.

That said, given what we know about unconscious bias, it is important for reviewers at least to be self-reflective about this. Which I understand some are not very inclined to do.


It seems to me that this faculty member has it all wrong. (I'm not sure exactly how much of this the anonymous commenter believes, and how much s/he is merely attributing to the faculty member.) Blind review goes in two directions, and this means that the referee is not supposed to know whose paper it is. Because, as the anonymous commenter notes, there are loads of unconscious biases, and blind review is supposed to control for them. But it is not enough to be "self-reflective" about this. If the biases are unconscious, it is literally not possible to correct them via self-reflection. The way to correct them is to eliminate the bits of knowledge they operate on. And so the effective way to be self-reflective about latent biases is to acknowledge that they are there, and to realize that blind review procedures are the only way to protect against them, and to observe those procedures.

Or am I missing something?

--Mr. Zero

P.S. The faculty member is right about how the author shouldn't know who the referee is.

8 comments:

Anonymous said...

Well, I think the faculty member's comments could be charitably interpreted in the following way: suppose you are refereeing a paper in the phil of language that defends a certain famous line of argument developing it in a different direction. However, the author never gives credit to the famous person we all know is responsible for that line of argument. Guess who? This kind of situation is unavoidable, of course.

Mr. Zero said...

I see what you're saying, anon 7:23. But I can't imagine that this happens "very often." I haven't been around the block as many times as this senior faculty member, but I would imagine that the occasions on which you can make a positive ID just from reading the paper would be rather unusual, and uncommon.

zombie said...

I think author identity would be, at best, underdetermined in most cases. One might make an educated guess, and be right. But evaluating the paper on the basis of that guess would be a violation of professional ethics if there is an expectation of anonymity.

Popkin said...

The internet has given me the impression that philosophers frequently agree to referee papers where they know who the author is (and that journal editors frequently permit this). This is obviously a huge problem, and any referee who thinks he/she won't let that knowledge influence his/her decision is kidding him/herself.

Another thing that's really aggravating about that thread is the discussion of sending R&Rs out to new referees. When a journal sends a revised paper out to a new referee, the process is beginning again from scratch, and so the journal is wasting the author's time (the time spent revising the paper in response to the first referee reports).

Jonathan Ichikawa's suggestion that this practice doesn't waste the author's time because good referees are likely to have the same concerns about a given paper strikes me as particularly implausible. I don't think I have ever had two referees make the same complaints or raise the same objections regarding a given paper. When I get multiple referee reports on a paper, typically there is no overlap between the criticisms/objections raised by each referee (and it's not because the relevant referees have done a bad job).

Anonymous said...

re: Mr Zero at 7:30. I work in a specialized area and i would say that I can identify the authors of about half of the papers I get sent to review. It's not always completely conclusive but a surprisingly large fraction are papers I have read in draft form because the authors sent them to me for comments. Other papers I can identify the author by the distinctive views they have defended in prior work. For journal editors there is a real dilemma since the potential referees who are most knowledgeable are the ones who are least likely to be able to provide a blind review.

My practice when blinding is compromised is to refuse cases where I'm not confident I can be unbiased, and to leave it up to the editor (I inform her that my review can't be blind) in the others. I don't discount the possibility of unconscious bias but on the other hand I also think there is some advantage to having articles reviewed by knowledgeable referees and it's sometimes not possible to meet both goals.

Mr. Zero said...

anon 10:19,

I'm picking up what you're setting down. But I think we're on the same side. You can identify these authors because you work in an area with very few other people, so there are not very many possibilities. This is pretty unusual, right?

But more importantly, you see knowing the identity of the author as a drawback. Although you think it can be outweighed in some of the the cases you've been involved in, you inform the editor of the situation and let the editor make the call. This makes it seem like you disagree with the senior faculty member I mentioned, who seems to believe that the important thing is that the author not know who the referee is, and that the referee knowing the identity of the author is no big deal.

Now, you seem to suggest that you inform the editor only if you suspect that you cannot be unbiased. I would suggest that you probably ought to inform the editor in all cases. The problem with latent biases is that you don't know about them. Since you might be biased without being aware of it, you should probably let the editor make the call whether you think you're biased or not.

At least, that's how it seems to me.

Anonymous said...

Mr Zero, anon 10.19 says that he/she refuses papers where he/she cannot be unbiased and informs the editor in the cases where he/she knows the author but thinks he/she can be unbiased.


-=

One issue that must make one's identity more likely to be recognised by the reviewer is when one refers to one's own previous papers in the third person. If one refers to and builds on those papers then that does not necessarily mean that one is the author, but it is does make it quite likely...

Mr. Zero said...

Mr Zero, anon 10.19 says that he/she refuses papers where he/she cannot be unbiased and informs the editor in the cases where he/she knows the author but thinks he/she can be unbiased.

Whoops. Sorry, anon 10:19