Wednesday, April 21, 2010

Many to One

Tyler Cowen posted an interesting thought on his blog Marginal Revolution:
The Back-up Plan

The journals of the American Economic Association have started an experiment that acknowledges the reality that papers move from one publication to another -- and the system could save authors considerable time, and publications money. In the experiment, authors of papers that are rejected from the flagship journal -- American Economic Review -- can now opt to have referee reports sent directly to one of four other journals published by the association.

So far it looks like a near-Pareto improvement. Here is more detail; by the way, editors from sociology and anthropology say that plan wouldn't work in their disciplines, though neuroscience has a reviewing consortium.

On the one hand, I like the idea that I'd only have to wait on one reviewer to hear back from several journals. On the other, I'm not sure that I want one reviewer to have that much influence over whether my paper gets accepted at multiple journals. It also seems like this system would affect the way the review reads the paper - this a a Journal A paper, this is a Journal B paper - and I don't have a good sense of whether I'd want that or not.

Which is all just to say, any thoughts about how to fix our busted system seem worth entertaining.

-- Second Suitor

19 comments:

Anonymous said...

It's not entirely clear what the problem with our system is. I've just been checking to see if the refs have finished with any of my papers yet. They haven't. I just checked on four pieces that are under review for 4+ months now and because I can follow their status online, I know that the holdup is not at finding a referee but getting the referee to do his job. That sucks, I refereed four things in January. Not one remained on my desk for more than two weeks. I'm probably busier than most of my refs. Part of the problem is just bad citizens not doing their share of the work.

Why would you let a second revise and resubmit just sit on your desk for four months and do nothing? (Wait, probably because you're the kind of referee who takes forever to do anything and writes referee reports where you raise an objection, admit that it was addressed successfully, but then find a new objection that had been addressed in the first draft in a passage that you said should be cut to streamline the paper.)

Meanwhile, I'm a tad annoyed at Phil Studies right now. I can't complain about the length of time it's taken to get a verdict from them (6+ months) because I know that the referee is the editor. I had hoped for blind review and a normal referee who could be prodded by the editor if he didn't get his report in on time, but I didn't get either.

Ben said...

I submitted something to the Journal of Medical Ethics and noticed that they seem to have a similar system (with British Medical Journal, etc), though there was nothing else that seemed appropriate for a philosophical ethics paper.

I agree that I wouldn't like a monopoly whereby one cranky referee can exclude you from all the journals, but something like this strikes me as a good diea. It must be terrible to almost make it in a top journal, like Phil Review - a rejected R&R say - and then have to start all over again...

Peter Borah said...

Note that in this case the "cranky reviewer problem" is not an issue.

Quote from the article:

"Moffitt noted that every part of the process is voluntary. The author who has been rejected sees the reports and decides whether to have them forwarded to the next journal. And the editor of that journal decides whether to use them, or go through the standard process (or to seek just one additional review."

Anonymous said...

Anon 5:49am is, I think, partially right. A big problem with the process is finding and motivating referees to do this work in a timely fashion.

I've never refereed for anyone important, just a couple grad conferences and the like, but I can see precisely why it is a problem. A person has so much time, and so many demands on it. If you think about what makes practical, professional sense for people to do, prioritizing pretty much everything over refereeing papers is clearly sensible.

Here's why. If you are applying for tenure, is your review committee going to give a good goddamn about your referee activities? Unlikely. Research activities, teaching evals, services provided within your institution are likely to be the things reviewed. The presence of some CV lines saying you were a referee are a minimally good thing, but the absence is unlikely to do you any damage. So it makes sense to prioritize all those things over and above refereeing.

If you are already tenured, is your annual evaluation by your department chair going to be made better because you refereed for a journal? Unlikely. What good does refereeing do for your career? It's not clear it does anything.

In other words, referees have absolutely no incentive other than wanting to avoid the label of "bad citizens not doing their share of the work." But why should referees do it for free, or for no advantage at all, when they could make use of their time doing things that have more strategic benefit, when it comes to their career?

Forgive me for sounding a tad cynical but I could not give a damn if I am a "bad citizen not doing my share of the work" when I don't do certain things, because the consequences for being a "bad citizen" are pretty much naught, but the consequences of not doing well enough with my research activity, teaching, and committee activity on campus are concrete -- I will get poorer evaluations, that could affect my earnings or my long-term career potential. If this kind of volunteer work really is so important to academe, then academe has a really odd way of showing it by affording it absolutely no value in the system of recognition and merit.

Like I said, I've not done any work with major journals, though its certainly on the horizon for me. And I'm sure this has been suggested before, but does anyone think that if the journals paid referees, even a very modest sum, to turn around articles for referee within a certain time frame, it would accelerate the process? This is why I think this might be more a problem for the editors and the managers of journals. If they wanted people to move quickly, they'd find a way to make them move quickly, but they don't, so its obviously not their priority to get things reviewed quickly. Publishers would probably rather you wait six+ months than fork over some of their profits to pay referees a piddly stipend.

Perhaps the real way to solve this is to take journals out of the hands of publishers and put them into the hands of philosophers by using the internet as a forum. Developing a website that performs the same function as a journal would be pretty cheap/easy, and we could preserve the exact same process and standards and cut out the middle man (and possibly the charges for access).

Just some thoughts.

Anonymous said...

"If this kind of volunteer work really is so important to academe, then academe has a really odd way of showing it by affording it absolutely no value in the system of recognition and merit."

I don't think it's crazy to think that members of the philosophical community who benefit from the good work that some referees do and rightly resent bad referees can be expected to act like a good referee. What's in it for you? Nothing. What happens if everyone is like you? No refereeing gets done. Glad you haven't been tapped to sit on one of my papers doing nothing.

Because it's my day off, I'll check the various sites between 3 and 4000 times to see if there's any progress. I've been told by a senior adviser that refs are slower during the summer. You know, because it's vacation time. Of course, they're also slow between semesters. You know, because it's vacation time. Oh, and they'll be slower still once the school year kicks up again.

JMc said...

Has anyone here submitted articles to journals that have submission fees (and then "expedited submission" fees)? I was taken aback when I saw this and so moved on to another journal.

What are others' thoughts on this practice? Was it a good reason for me to move on to another journal insofar as being indicative of a poorer quality journal?

Anonymous said...

Here's an idea: How about basing tenure decisions on quality, not page count? Let's offer the option of having one's tenure decisions based on our top three articles. I know that I've published too much passable material (usually in good places too) purely to satisfy the criteria for tenure. In so doing, I've helped clog the referee system and the reading burden of colleagues in my sub- specialty with unnecessary text. I'd say that I've written maybe two excellent articles, the rest... meh

We'd need some of the fancier departments to lead the way in a move like this. I don't think deans and provosts would have a huge problem with such a move.

zombie said...

"Forgive me for sounding a tad cynical but I could not give a damn if I am a "bad citizen not doing my share of the work" when I don't do certain things, because the consequences for being a "bad citizen" are pretty much naught, but the consequences of not doing well enough with my research activity, teaching, and committee activity on campus are concrete..."

Ah, but see, there ARE bad consequences when referees don't care about being bad citizens, and those consequences are bad for you. Free riders can't rationally expect to get away with free riding indefinitely. The consequence is that there are not enough good referees to get the work done in a timely manner, which will not only hurt you when it comes your turn to submit a paper, but ultimately hurts the profession, if good papers go unpublished.

So, from both a social contract and a consequentialist perspective, it is in your interest to be a good referee.

From a deontological perspective, you have a duty to treat others with respect, and that includes giving their work the consideration it is due.

So, however you want to slice it, you have an obligation to read the papers you have agreed to referee, and to do it in a reasonably timely manner.

Mr. Zero said...

referees have absolutely no incentive other than wanting to avoid the label of "bad citizens not doing their share of the work."

For one thing, that's a soft way to put it. Another way to put it is that when you have your paper refereed, you accumulated a debt. You acquire a duty to do the simple work of refereeing papers in return. If you don't do it, even though you should, a less gentle person--not me, mind you--would say you're an asshole.

If this kind of volunteer work really is so important to academe, then academe has a really odd way of showing it by affording it absolutely no value in the system of recognition and merit.

That's like wondering what the point of not littering is, when nobody gives you a medal for not littering. "If it's so important not to litter, where's my medal? Where's my reward? What's in it for me (other than the maintenance of a smooth-running society from which I benefit in obvious, basic, and crucial ways)?"

herbhart said...

I pretty much agree with Zombie and Zero, but I do think Zero's last remark might be a little harsh. It's a lot of work to do one's share of refereeing, and let's face it, some people don't. That means those of us who do are at a professional disadvantage (since there's no reward), and that does kind of suck.

In my annual CV update (the one that goes to the Dean), at the line where I list the journals I've refereed for, I add a sentence pointing out that refereeing is an important job that takes a lot of time. This probably makes no difference whatsoever, but it makes me feel a little better.

Anonymous said...

I consider refereeing papers in a timely manner to be simply part of doing my job. Just like grading student papers in a timely manner is part of my job. I always referee and grade papers promptly because I like my job.
When I assisted someone famous on a top journal we kept a list of philosophers to be avoided at all costs since they were so hopelessly slack.

Bryan said...

I've recently been thinking about this too. Some suggestions:

- Online interface for anonymous author-reviewer communication.
- Monetary incentives for high quality reviews. (I'd be willing to pay a submission fee for a high quality review!)
- Stop printing paper copies of journals.

Mr. Zero said...

It's a lot of work to do one's share of refereeing, and let's face it, some people don't. That means those of us who do are at a professional disadvantage (since there's no reward), and that does kind of suck.

I see what you're saying, but I don't see why you think it supports the claim that I was being too harsh. Doing your fair share of refereeing puts you at a professional disadvantage only in comparison to people who don't do their share. Their advantage is the advantage of cheating. People who cheat, and who thereby put honest people at a disadvantage, are assholes. And if nobody cheated, nobody would be at a disadvantage by not cheating.

herbhart said...

I guess what I really meant to say is this.
Yes, of course we have to do our share, since we benefit from the 'fair system of cooperation', and even though it takes a lot of time, it sure beats committee work. But around this time of year, when I pay my taxes, it just irks me that some people don't and they get away with it. Grr. I think, "Couldn't somebody do something about that? Fix the incentive structure or something?" And that's a bit how I feel about refereeing. Right, I shouldn't get any big reward for doing it, but maybe the free riders who do none could get a little kick in the pants or something.

Anonymous said...

Anon 7:33 here.

Allow me to make some clarifications and amend my position.

-I myself have neither had papers (not?) refereed by any journals nor have I failed to referee any papers which I have committed to referee (which as I mentioned, have all been for grad conferences and the like), so although I'm a free-rider sympathizer, I'm not one myself. So I'm not holding up anyone's work (yet).

-Look, I don't disagree that the referee system is a good idea in principle. Ideally, we are all working together to enhance the philosophical sphere, produce good research, and help each other along in the search for knowledge and wisdom. And that's all well and lovely in academic utopia.

But when people's livelihoods depend on external standards of evaluation (like those set forward by academic institutions for tenure, promotion, merit pay raises, etc.), those standards of evaluation play a significant role in people's deciding what to do.

As much as I'd like to read and comment on the work of my peers, and help them along, if I have to choose between doing that, which my tenure review committee is not going to care about, or doing a bunch of other stuff that's going to make my tenure review committee all smiley and joyous, I'm going to choose the latter every time.

So, Zombie, yeah, the academic system creates the free-riders by not rewarding behavior that should be rewarded. It's basically subversion from within. Once you reach a certain critical mass of free-riders, the system breaks down. But the problem is that it is always most rational to be a free-rider. Prior to the breakdown, being a free-rider means getting the benefits and doing none of the work. After the break down, doing the work only benefits others (my quick refereeing will not make other referees act more quickly and get my papers back to me sooner).

The system only works with no free-riders. What that means to me is that the current system is unsustainable, and, yeah needs to be somehow revamped to prevent subversion, either by rewarding cooperators or punishing defectors. Telling defectors they are "bad citizen" is form of sanction that has absolutely no teeth, which is what I meant to say when I said, "Forgive me for sounding a tad cynical but I could not give a damn if I am a "bad citizen not doing my share of the work" when I don't do certain things, because the consequences for being a "bad citizen" are pretty much naught, but the consequences of not doing well enough with my research activity, teaching, and committee activity on campus are concrete -- I will get poorer evaluations, that could affect my earnings or my long-term career potential. If this kind of volunteer work really is so important to academe, then academe has a really odd way of showing it by affording it absolutely no value in the system of recognition and merit." Consider my position amended, or clarified, or whatever it is I've done.

-Also, I don't think its necessarily fair to blame people for having to be practical when it comes to making choices about what to do with their time. We live and work in a fairly cutthroat environment, where every thing we do or don't do can affect our professional futures and well-beings. In a system that relies on cooperation but issues benefits and rewards defection, we will always see this kind of a problem. It's not as though I'm personally advocating that everyone who referees in a timely fashion stop doing so. I'm just pointing out that there are understandable reasons for people's acting the way they do, at least in some cases.

Anonymous said...

"-Also, I don't think its necessarily fair to blame people for having to be practical when it comes to making choices about what to do with their time."

It's hard for me to imagine that these are the choices. I refereed four papers, a book manuscript, and a textbook proposal this semester. I was able to get referee reports in while teaching 4 courses at one school and two at another while publishing myself out of that crap situation. If you can't get tenure without shouldering a wee bit of the refereeing responsibility, that's really, really sorry.

zombie said...

Anon 7:33, I'm not sure you get what I'm saying. You can't say that you are too busy doing your own research (which I assume includes trying to get published), while simultaneously making it impossible to get published by neglecting to referee papers. If everyone does that -- says they're too busy trying to get published to referee papers -- then nobody gets published because nobody referees. That's when the system falls apart. But if you are getting published, then it is obvious that somebody IS refereeing papers. If that's the case, then you are breaking the social contract, and you are using others as a means to your ends without respecting them as ends in themselves. And even Kant knew that behaving in that way would ultimately be self-defeating when you universalize your maxim ("It's OK for me to neglect to referee papers as long as my excuse is that I'm too busy trying to get my own papers published"). You are shooting yourself in the foot (in addition to exploiting others).

I guess I'm naïve, but I'm shocked at the self-justifying, unethical behavior of philosophers who exploit a system when it works in their favor (i.e. when they can be free riders), then complain that the system makes it too hard for them to do their own part to keep it working for everyone else. Baloney. We've all got too much to do, but refereeing papers is part of the job. I spend several hours every week reading and critiquing a colleague's paper. It means I have to put my own work aside for those hours. But I also think it makes me a sharper, better philosopher, and helps the colleague as well. And I have my own research to do, and my family to take care of, and all the million other things that suck up my time. I just cannot generate any sympathy for The Foole who lets some anonymous sap's paper gather dust on his/her desk. If you're not going to referee papers, then SAY you're not going to referee papers, and deal with the consequences.

Anonymous said...

But the problem is that it is always most rational to be a free-rider

Yes, every time I do my bit for the common good I think to myself how irrational I'm being.

You might want a different theory of practical rationality.

Anonymous said...

It's worth noting that the Neuroscience Peer Review Consortium is voluntary - authors choose whether to have their reviews forwarded to another journal.