Wednesday, April 28, 2010

The Best Idea Anyone Has Ever Had About How to Get People to Referee Better

There's been some discussion lately about how to get people to be better at refereeing. Referees are a vital part of the profession, and they're an all-volunteer corps, and they are, by and large, not awesome. It takes tons of time to get your paper refereed, there are often no helpful comments, and there is a strong temptation to free-ride.

An idea that gets bounced around maybe once a year is charging authors a submission fee and then use the funds raised to compensate the referees. This will give the referees an incentive to do their jobs and have the side-effect of reducing frivolous submissions. This guy, whose wikipedia entry suggests that he is a pretty accomplished person and probably not a dumb-dumb, suggests a sliding scale based on desired turn-around time. He says, "a submitter's willingness to pay a higher fee, some economists might say, serves as a signal of the submitter's confidence in the quality of the paper."

Now, I'm no economist, but I would have thought that a willingness to pay a higher fee under such a system would serve most straightforwardly as a signal of how much of a hurry the submitter was in. I would have thought that there would also be a significant relationship between a willingness to pay a higher fee and the submitter's access to disposable income. I'm sure that confidence in the quality of the paper is on the list somewhere, but it's not right at the top or anything. (Not that author confidence of quality is a super-reliable indicator of actual quality, anyways.)

This is the fundamental problem with all such proposals: they disadvantage graduate students and junior faculty, who don't make as much money as senior people, and for whom the marginal utility of an extra publication is greatest. That is, such proposals are unfair.

So here's the idea. Charge people a fee. But the fee isn't for submission; it's for publication. You wouldn't want just anyone who submits to pay this fee. And the fee isn't monetary, so it's not taking food out of the mouths of Professor Newbie's kids. The fee is the job of refereeing itself. The journal accepts your paper and part of the permission-to-publish contract includes an agreement that the author serve as a referee for that journal. There is wiggle room concerning how many times would be fair. But I assume it would be at least two, and maybe more. The jobs could be spread out over a year or two. And the journal would have some leverage over its referees, since it could withhold the paper until (some subset of) the author's refereeing duties had been discharged.

Does this proposal have some problems? Absolutely. But I submit that if it's not better than the current system, nothing is.

--Mr. Zero


Bryan said...

Another idea I've heard floating around: publish the names of the referees with the paper. Ever notice the average quality named blog commenters, as compared to the anonymous ones? Plus, good referees often bring significant improvements to a published work. It's only fair to give them some credit for this service.

Anonymous said...

I think you've missed Tushnet's point about signaling confidence. (I read the appearance of "though" in Tushnet's penultimate sentence as a hedge.) A willingness to pay a higher fee sends the signal that you'd like the referees to finish sooner rather than later. Now, if you were confident that the paper was a good one (and therefore likely to be accepted), you might not care that much if the referees took a little longer. (Unless you wanted to get the pub on your cv before turning in your tenure dossier.) In the end, you think the paper's likely to get published, so you're not anxious to send it out to another journal.

However, if you weren't so confident in the paper's quality (and therefore less sanguine about the prospects for acceptance), you'd want the referees to finish up reasonably quickly so you could get the paper back in the pipeline sooner rather than later.

So, if you're confident in the quality of the paper, you're less likely to be in a hurry. Thus, you're less likely to pay a higher fee. So willingness to pay a high fee sends a signal about confidence, but not a positive one.

zombie said...

I had kinda thought something like that was already in place. You're asked to referee once you're on the list, so to speak, as a published author. But that is a more informal arrangement, to be sure, than the one you propose.

I like your idea. It makes sense. I wonder if there's a better way to structure the disincentive -- especially since it can take so long to get published as it is.

I also think it would go some way to solving this problem if more junior scholars were tapped for the job. Say, Professor X gets a paper, and with the permission of the journal, hands it off to a well-qualified grad or postdoc. Those are scholars who could use the line on their CV, which might be an incentive to do it. And since so many of us are unemployed, it will give us something to do other than whittle busts of Camus out of soap.

Anonymous said...

1. I've never been asked to referee for any journal I've been published in - and I've been publishing 7-8 years. So the system suggested isn't 'informally in place'. I have been asked to referee, several times, for a journal I'd be highly unlikely to consider to submit to.

2. Here's my suggestion fro improving the dire refereeing situation. Don't give tenure to people who haven't shown that they're willing to take on their fair share of refereeing.

3. I'm no economist, but I would have thought that a willingness to pay a higher fee under such a system would serve most straightforwardly as a signal of ,how much disposable income the submitter had.


Anonymous said...

Bryan, not sure how this helps with referees who don't do a good job, though. It only rewards those who did - and may unjustly reward those who didn't.
(verification word: prolve!)

Anonymous said...

I am skeptical about the idea of fees. One problem is that, in some settings, a purely social sanction has been shown to be more effective than a financial one (see behavioural ecos literature). Given that the fee involved will, presumably, be quite low, this could be an issue.

I would like a variant of the OP's scheme: Set up a central database of who has refereed and how often. If you are in 'debt' beyond a certain point your stuff does not get refereed.

Such a database could be tweaked in all kind of ways: there could be a rule that you may never get your reports back faster than your own average turnaround. I think there will be a remarkable improvement in turnaround times under such a system. My guess is that average referee turnaround will drop to less than a week.
As we all know, papers take n months plus 2 days to be refereed anyway.

Anonymous said...

Just a quick reply to zombie:

Getting to be a referee is a bit more complicated (weird?) than that. For example, I got a request to referee a paper for a journal that had rejected my only submission. On top of it, they rejected the paper 5 or 6 years prior to the refereeing request, and in the interim I'd moved to two different jobs. (So, I was probably a bit difficult to track down.)

In contrast, journals that have accepted my work haven't asked me to referee.

Anonymous said...

It is a fantastic idea. Perhaps combined with a joint pool/database as suggested by anon 11:46 (run by... the APA??? this would require some doing) because not everyone referees for the journals they publish in, or gets to referee as much as they need/want/get to publish. Or simply because it's not always correlated in volume over time (one year you send out 4 papers and referee one, the next it's the opposite). Yup; the best idea anyone has ever had, Mr Zero.

Anonymous said...

Somebody has hit the nail on the head ... obliquely. Referees should act professionally and if that is not happening in the volunteer scenario, then referees should be professional referees. In this line of work, that would mean at least "professional philosophers" who want to earn some extra pocket change doing the grunt work nobody seems to like to do as a volunteer. The whole system would become more vibrant all the way around. How that's funded, who knows. Perhaps the journals who currently get the backbreaking work done by volunteers should come up with a way. Maybe some foundation could endow that. Clearly this process IS in need of some market forces.

zombie said...

Anon 5:30 -- interesting.

So then I have no idea how journals decide who referees. Maybe that's part of the problem -- the work is not distributed rationally or fairly.

Anonymous said...

Zombie at 9:31 comment is great. That and some pittance of filthy lucre for the task would brighten spirits all the way around.

Popkin said...

Anon 5:09 is ignoring the important point made by Anon 11:46. Injecting "market forces" by paying referees is likely to make the problem worse by changing the nature of the transaction and making it easier for people to justify putting off doing the work (think of that well known story about the daycare that started charging parents for picking up their kids late, only to find that this exacerbated the problem).

I say there needs to be some way to publicly shame negligent referees. Maybe a website that listed how often people agreed to referee and how often they met the journal's deadline.

Anonymous said...

"So then I have no idea how journals decide who referees. Maybe that's part of the problem -- the work is not distributed rationally or fairly."

Agreed. I floated the idea before of having a central place where people can put their CVs and AOSs into a database where editors can search for referees. I'm not technologically savvy, but I think this should be done. Case in point. The editor that breaks blind review to review things on his own is likely doing this, in part, because of an insufficient stock of referees.

holyoke said...

Um, considering academic journal publishers are making money hand over fist already, make them pay the referees.

You're forgetting that - outside of the academic world - people get paid to write, not the other way around. Throwing yet more money at fatcat publishers is akin to punching yourself in the face.