Thursday, May 30, 2013

Peer Review Review

From a variety of sources, I have learned about this not-at-all recent study. From the abstract:

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables. 
The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices. 
With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.
 While I don't want to get carried away, or anything, this would seem to be a somewhat serious indictment of the peer-review practices of the psychology journals punked investigated by the researchers. Looks bad. Best case scenario is, publishing in these journals is a crapshoot where the odds are an abysmal nine-to-one against; worst case scenario is we're all getting butchered.

However, I was a little disappointed that the study didn't include something in the way of a control group--it seems to me that it would have been a better design if they'd substituted (actual) prestigious institutional affiliations for some of the articles, instead of using all fictitious institutions. And, along the same lines, it would have been a better design if they'd have given the same treatment to a group of papers published in C-level journals--submitted them to high-level journals, half with affiliations with fictitious institutions and half with affiliations with real, prestigious institutions. It seems to me that a study with that design would be a lot more conclusive. (Not to say, 'conclusive.')

(Also, a commenter at Philosophers' Cocoon says that the journals investigated all practiced non-blind review procedures. I'm working from home today, and am unwilling to jump through the hoops I'd need to in order to read the article, so I'm just going to take her word for it. But if that's right, it takes almost all of the "wow" factor away. It's still kind of bad that they didn't recognize the articles as having been already published by them, but if your job is mostly to receive submissions, send them out to review, and deal with the results, it's easy to imagine that you wouldn't catch on to something like that. I, for one, wouldn't be on the lookout for it.)

--Mr. Zero

31 comments:

Anonymous said...

Different but somewhat related issues are discussed in Stephen Hilgartner's fascinating essay "The Sokal affair in context." Science, Technology & Human Values 22.4 (1997): 506-522. (esp. in connection with Epstein's piece -- i.e., Epstein, William M. (1990) "Confirmational response bias among social work journals." Science, Technology, & Human Values 15:9-38.)

Anonymous said...

I just received a rejection from the best journal in my subfield. There was no report just a two paragraph note from the editor that made it clear to me he didn't bother to read the article and my pedigree wasn't good enough to stain the pages of his journal.

appropriately, the confirmation words are "forum bewailing"

Anonymous said...

5:10:

What was the journal, and what was the note?

Anonymous said...

5:10 here:

So there's a reason I opted for anonymity. I will say this, this was a journal in a specialized field in philosophy so not JPhil or anything like that. The editor is pretty well known in this field. The paper addressed a somewhat ongoing debate that has had some articles published in said journal recently. Despite that I was told that my paper wasn't right for said journal and then given advice to look at material that was irrelevant to the main argument and showed that he had barely bothered to skim it. My point was that before papers even get to reviewers they have to go through an editor (and this part is of course not blind). The editor did not recognize that this was a debate in his own journal (despite the frequent references to those pieces) and suggested I pursue a line of inquiry that has nothing at all to do with the problem I am addressing.

Anonymous said...

I feel for 5:10!! I am sure a lot of us do. I teach at a crap institution, I don't even bother to send my work to top journals anymore. In my last experience I received a quick rejection from a top journal based on a single 2-line referee report: the paper was good, novel thesis bur one section of the paper was unclear -- and the editor decided rejection on that basis (I suppose there was no explanation just an outright rejection with the two sentences pasted in) It felt like my submission was treated as little more than a joke.

Anonymous said...

From my own experience, peer review is a corrupt joke. Let me count the ways:

1) Most recently I had a revise and resubmit, which I did according to the recommendations provided. The paper was later rejected, but the comments of the latter reviewers made it clear—and I mean apodictically clear—that the imbecile editor had sent the original draft of the paper out again instead of the revised version I had spent a month working on! In all this wasted almost a year of my time (as well as that of the reviewers).

2) I had another paper rejected because the editor complained that I didn't cite "the most recent paper" on the topic. Except that I did—three freaking times. Also, the paper in question was written by him.

3) I've had two papers that were effectively "lost" in the review process, which I only found out about months later when I emailed the editor asking if there had been a decision.

4) The notion of "blind review" is, in the age of Google, absurd. I could provide personal examples, but I don't know how to do so without outing my identity.

5) Lastly, like Anon@5:10, I've gotten several reviews that made evident that the reviewer did not bother to read my paper. See #2 for one example. In a different paper, a reviewer chided me for neglecting a key quote—except I did in fact include/analyze the quote at length and in the body of the paper...albeit on page 4, which I guess he/she never got around to.

I could go on, but I'd rather not. It's too upsetting.

The point is, peer review is intended to serve as a gatekeeper for the discipline. It does that in a certain sense I suppose, but not in any way that strikes me as positive...more in the way a gated community serves to keep out the riffraff.

Anonymous said...

"My point was that before papers even get to reviewers they have to go through an editor (and this part is of course not blind)."

I realize this thread isn't about desk rejections, but could someone explain to me why there are blind reviewing policies at journals intended to thwart biases based on having knowledge of the author's identity and affiliation and also the practice of editorial desk rejection? It seems like one hand giveth what the other taketh away.

Anonymous said...

From quoted abstract: 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices

Not too much jumping through hoops necessary.

Anonymous said...

The fact that all the journals have only single-blind review processes means that the results don't tell us, as philosophers, much about our situation. However, single-blind review (referee knows author, author doesn't know referee) is the norm in most of the natural and social sciences. I don't know why, but it is. And this study clearly shows that single-blind review does not accomplish the purposes peer review is supposed to accomplish. What it doesn't tell us is whether (or rather, how well or poorly) the system we use in philosophy works.

Anonymous said...

3:00 - Some journals do practice 'triple blind' review, where the person who decides (a) whether to send the paper to full peer review, and (b) who the referees should be doesn't know the identity of the author. Pacific Philosophical Quarterly, for instance, practices triple blind review (with (a) and (b) decided by a USC faculty member or advanced graduate student with relevant expertise, and not by the managing editor who corresponds with the authors). Of course someone needs to know the identity and affiliation of the author to ensure that the paper doesn't get sent to the author his/her self or one of his/her department colleagues, but that person (the managing editor) only decides among the short list of referees recommended by the 'blind' internal reviewer. At PPQ, the internal referee (who knows the identities of the external referees, but, again, not of the author) also is involved in deciding what to do once the reports come in, in case the reports don't make that decision obvious.

All journals (or at least all journals that ever issue desk rejections - last I knew, Australasian Journal of Philosophy was sending literally everything out to peer review) ought to do this, though it's a little more work than doing it the other way. I think the main reason people think it's ok to cut corners here is that they think there's less room for bias to have an effect in distinguishing obviously unpublishable papers from possibly publishable ones than there is in the later 'publish or not' decision. However, (1) as commenters have already mentioned, some journals issue desk rejections to papers which are definitely not obviously unpublishable; and (2) there are going to be borderline cases here just like everywhere else, so the claim that there's less room for bias is probably false.

Anonymous said...

My point was that before papers even get to reviewers they have to go through an editor (and this part is of course not blind).

This definitely depends on the journal. At Ethics, for example, the managing editor makes sure that a fully anonymized paper goes to the Editor for desk review. So if it is rejected at that stage, it is not because the first decisionmaker at Ethics knows the pedigree.

Anonymous said...

"Pacific Philosophical Quarterly, for instance, practices triple blind review (with (a) and (b) decided by a USC faculty member or advanced graduate student with relevant expertise, and not by the managing editor who corresponds with the authors)."

This is very useful to know. I'll have to send them my work in the near future.

Honestly, we should come up with "rankings" of journals from the perspective of authors, not the twaddle Leiter puts up. I know I've had good experiences (not the same as acceptances) and also very bad experiences. It would be interesting to see how they match up.

zombie said...

Then there's this:

http://researchnews.osu.edu/archive/matilda.htm

Study shows bias against women scientists. Abstracts with female authors were judged more harshly than the identical abstract with a male name.

So, all kinds of bias. For which the obvious solution is triple blind review.

Anonymous said...

Anon. you claim that Pac Phil Q. does triple-blind review. I understood them to run PPQ along the model of other pedigree-driven house journals in our field. According to the website

"A submitted manuscript first goes through a substantive inter- [sic?]departmental initial review which usually takes 2-3 weeks. If it passes the review, it will be sent to a referee or referees who will be given 6 weeks to provide a report. If the report is positive, the editors will take anywhere from a few days to a few weeks to make the final decision."


Andrew Cullison's survey site indicates that PPQ does not live up to these time promises btw.

I'm not trying to pick on PPQ and will drop my skepticism if you can point me to a policy statement where they commit to triple-blind review.

Also, it's true that AusJP does really great work and is a model for good practices in the field, it's not true that they send everything out for review. They do their share of desk rejects like every journal in our field.

Referees are a precious resource and good editors don't overburden them with papers that are unlikely to make the cut.

Anonymous said...

"Honestly, we should come up with "rankings" of journals from the perspective of authors, not the twaddle Leiter puts up. I know I've had good experiences (not the same as acceptances) and also very bad experiences. It would be interesting to see how they match up."

This. Yes please.

elisa freschi said...

Does not this case, together with the ones of plagiarism (see here: http://www.newappsblog.com/2010/09/on-plagiarism-and-the-refereeing-institutions-of-philosophy.html), suggest that there ARE valid reasons for googling some lines of an article you received for review, in order to? (After all, most people do it with papers they have to grade, in order to check for plagiarism). And how to accommodate the need to detect attempts of plagiarism (even as extreme as resubmissions!) with the need to be as blind as possible in order to avoid unconscious biases?

Anonymous said...

Submissions to conferences are not blind. Applications to jobs are not blind. Blind review of articles is both impossible and over-rated.

Anonymous said...

Off topic question: If one was asked to pay for the plane ticket for a campus visit, what is a reasonable amount of time to wait for the reimbursement. Thanks!

Popkin said...

Referees should not be googling text from papers to look for plagiarism. Since you can't remain ignorant of the author's identity if you are googling text, presumably the journal does not want you to do so. Consequently, any responsible journal will have some other procedure for detecting plagiarism (although, it might be such an uncommon problem that most journals don't worry about it; it's certainly a less serious problem than the influence of various biases that unfairly affect what gets published).

elisa freschi said...

Popkin, perhaps a solution might be NOT to look on google while evaluating a paper and THEN haivng a second person googling a few lines of it after the peer reviewer has accepted it?

Anonymous said...

Some submissions to conferences are blind and perhaps job applications should be blind. There is a persistent prejudice for those who are privileged in this discipline that undercuts any rumblings of social justice.

Popkin said...

Elisa: yes, I think something along those lines should be the norm. The main thing is for the googling to be carried out by someone who has no influence over the decision about the paper.

Anonymous said...

"perhaps job applications should be blind"

Seriously? Let's assume this is desirable; how does it play out? The applicant can't note where she earned her PhD, nor can she list the members of her dissertation committee. Maybe she can list the names of her letter writers, but most people will (probably correctly) assume that if the majority of her letter writers are at one institution, that's where she is earning her PhD. So to be truly blind, letters of recommendation need to come in blind, too.

Previous publications and conferences need to be stricken, as that might also give away the applicant's identity.

So what are we left with? A list of courses taught (but without knowing where they were taught, to ensure true blindness), and a writing sample (which would have to be unpublished work, to ensure blindness).

Because far too many applicants will have pretty similar (because now completely bland) teaching resumes, hiring decisions boil down to focusing almost exclusively on an unpublished writing sample and the description of the full dissertation.

Yeah, you're right. All decisions should be made blind. Because the only way we can make the best decision possible is by ensuring we have the least amount of relevant information.

Glaucon said...

Blind? Blind. Blind, blind, blind, blind, blind.

Anonymous said...

7:46:

At my campus, it's customary to give the job candidate a check when he/she arrives for the on-campus interview.

zombie said...

7:46 - if you're asking, it has probably been too long. If this was for a campus visit during the usual early spring, it has been too long.

Anonymous said...

"Blind review of articles is both impossible and over-rated."

It's not impossible given certain parameters.

It is, I suppose "over-rated" if you're a well-connected crony looking to make up in nepotism what you lack in merit.

So congrats on that. It's all the rage these days.

Anonymous said...

"It is, I suppose "over-rated" if you're a well-connected crony looking to make up in nepotism what you lack in merit."

So congrats on that. It's all the rage these days."

Here's a question (and I am somewhat serious): why do we bother putting names on anything we publish? Why does it matter who wrote what article or book? Why should the review of a work be blind, but its consumption and use not be blind? Once the work is published, why does it matter who wrote it? And if it does matter who wrote it, then why shouldn't that matter in the process of review?

Mr. Zero said...

Hi Anon 12:35,

The point of blind review procedures is not that it doesn't matter who wrote the paper. There are obviously lots and lots of reasons why it might matter who wrote what. The person might want to use the paper as part of a case for tenure or promotion. The person might want credit for the ideas expressed. The rest of us might want to give the person credit (or blame) for those ideas. We might want to read other papers by the same author. We might want to be able to contact the author with questions, comments, or discussion. None of this has any bearing on the warrant of blind review procedures.

There are several important reasons in favor of blind review procedures. One is that reviewer anonymity protects the reviewer from retaliation of rejected authors. It also protects the review process from corrupt quid-pro-quo arrangements.

Author anonymity, which seems to be what you're really asking about, is important, not because it doesn't matter who the author is, but because knowing the identity of the author is unavoidably connected in the mind of the reviewer—who, like all human beings, is an imperfect cognizer—with all sorts of other information that is irrelevant and has no place in the decision-making process. These biases often do not act on a conscious level—you generally don't know you have them, you can't discover by introspection when they are or are not operating, and you cannot compensate for them by any conscious act of mind or will. The only way to avoid them is to deprive oneself of the information that triggers them. If you google "latent cognitive biases" you will find a large quantity of information.

I hope this helps.

Anonymous said...

"Here's a question (and I am somewhat serious): why do we bother putting names on anything we publish? Why does it matter who wrote what article or book? Why should the review of a work be blind, but its consumption and use not be blind? Once the work is published, why does it matter who wrote it? And if it does matter who wrote it, then why shouldn't that matter in the process of review?

This is a nice example of a false analogy. I'm going to save it for the next time I teach critical thinking.

Anonymous said...

For several years I helped run a grad conference in which review of submission was all done anonymously. I am well aware that journals get way more submissions than we ever received and that finding reviewers is a very challenging process etc. Nevertheless, I found seeing the anonymous reviewing practice in action instructional.

Most importantly,t at the grad conference anonymous reviewing was really really easy to run. All it required was to have someone whose only job in the reviewing process was to make sure submissions were anonymized. This is a boring task, made much easier by having requested that applicants submit papers anonymously. The anonymizer then sent the submissions to me. I found appropriate reviewers. All papers were reviewed at least twice. I should also point out that when I was running the conference I would go over the rejected papers to make sure that the reviewers had not been overly harsh or too dismissive. But again, the papers were still all anonymous.

The results were as follow: Over the four years I was involved we accepted roughly equal numbers of men and women; it looks like about half of our acceptances were from highly ranked schools (top 10 or so), and we accepted several papers from unranked schools. Overall, these don't seem like unreasonable splits.

THe whole process was a bit bureaucratic, but the anonymity introduced ONE extra step in the process, and not a particular difficult step. It's hard to not conclude from the simplicity of the process that journals without anonymous (double-blind) reviewing practices, don't want to have them. Again, I understand that journals have many many more submissions than I ever saw, and maybe there is some important disanalogy that I'm missing, but it's not evident to me.