Thursday, May 5, 2011

Crowd sourcing peer review? Free open access?

Anonymous brings this to our attention:

The idea is to create an open-access online philosophy journal (and then journals in other disciplines), with the peer review process crowd sourced. As many reviewers as want to read a paper can vote to accept/reject, with brief comments. Accepted papers will immediately be published online.

From what I can see, the open access will be free for authors. They are now recruiting reviewers.

Interesting idea.

Whaddaya think?



Anonymous said...

We won't know how well it works until it gets going, so sign up!

Nick Cameron said...

Looks pretty cool. Have put my name down as a volunteer, we shall see whether I am allowed.

Anonymous said...

what's the incentive to participate, either as author or as reviewer?

Anonymous said...

The WTF label seems appropriate. maybe I'm too old fashioned...

Anonymous said...

This is obviously a neat idea, and something everyone wants to see succeed.

But I wonder whether this incarnation can succeed. The single most important concern for junior scholars is tenure promotion and so forth. What is going to guarantee that a publication at Sympoze will help them achieve that goal? Is there any reason for people to believe the journal will be highly regarded in the near future? Are famous, well-respected philosophers backing it or publishing articles in the first issue? Is it housed and affiliated with a high-ranking institution? I don't seen any of this in the plans so far...

Anonymous said...

If I were reviewing a tenure application package, I would not consider this kind of publications to be equivalent to those which underwent peer review in "traditional" journals, where the editors and referees have been selected on the basis of known expertise in the field.

Anonymous said...

@Anonymous 9:55

Suppose that what you say is right, and untenured folks don't have much to gain from submitting to this journal. Then, assuming Sympoze does deliver on efficiency, it will become an appealing place for mid-career and later folks to submit work, since they will get quality feedback/decisions more quickly than with traditional journals. If the process is not only more efficient, but also does a good job of accepting good work and rejecting bad work, this journal will develop a reputation over time (as any new journal does), and eventually become seen as an acceptable venue for junior faculty to publish.

Anonymous said...

@Anonymous 2:32

Of course, but the problem is that the old farts don't seem to give a crap about speedy feedback, the Interwebs, Open Access, etc. They also don't tend to publish anywhere unless it's backed by a major institution, their famous friends, etc. Prestige can't be created out of sheer good will!

Anonymous said...

A better place to publish and receive feedback on your work is the new on-line journal American Dialectic:

zombie said...

Anon 7:47: The incentive as an author is to get quality peer review without the hassle and the wait (or so they say), plus open-access publishing without a fee. Additionally, they say, no more being at the mercy of an overburdened or biased reviewer, since you will have (potentially) many reviewers. Presumably, as well, this cuts out the editor as first-pass reviewer who knows who you are (or are not).

The incentive as a reviewer is (they say) that having a large, volunteer pool of reviewers takes some of the burden off of reviewers. You review what you want when you want. You do not have to write up a lengthy report to the editor. You either accept or reject a submission, and write a few brief comments for the author.

What I don't remember reading on the website is anything about copy editing or proofreading. If there's no one doing that, I can only imagine that the quality of the writing will be, to say the least, inconsistent (although maybe that gets taken care of in the reviewer comments -- but what a drag, as a reviewer, to have to proofread a badly written submission.)

Andrew Cullison said...

Thanks for this helpful feedback. I have been concerned about whether this will be recognized as equivalent to traditional peer review since I first started thinking about this idea. I hope to write up a detailed FAQ response soon.

I’ll note two things now. First, referees will not have access to papers outside of their AOSs, and we will be carefully screening applications for this. So screening for expertise won’t be absent from this process.

Second, I have and will continue to consult leaders in the field to help fine-tune the process with the goal of assuring that philosophers will treat papers vetted through Sympoze as having gone through a proper peer-review process. I am already in the process of putting together a board of advisors.

On a more enthusiastic note, I know this is a service that people want. Within the first day of announcing this, 40 philosophers (mostly tenured and tenure-track) volunteered to referee. I think we’re going to be able to make this work, and I think we're going to be able to address what I see as the primary worry.

Again, I'll post more on this soon over at the site.

Until then, please don't hestitate to send me any questions, comments, or suggestions you have. You can email me or post to this thread (I'll track it).

Oh...and please spread the word.

Anonymous said...

I do, all things considered, support this venture.
That being said, one concern I have is about nepotism, as a result of the publications being decided by a crowd (of qualified readers) vote. What I mean is, say A has friends B, C, and D whom also work in her field (or worse, grad students in a program she teaches at!). Most worrisome is the case where A alerts B, C, and D that she submitted an article which is now available for review, and B, C, and D go and vote for it to go to publication. More innocuous but still worrisome is a case where my writing style or views are well-known enough that its clear that I wrote the paper, and B, C, and D vote for it to go to publication.
I think these are surmountable worries, but worth considering.

Anonymous said...

Maybe Sympoze is the new Synthese.

More seriously, perhaps Leiter can pick up this issue, and see what lots of philosophers think about it.

Here's a NYT article on crowd-sourcing:

Anonymous said...

As persnickety as a single philosophy reviewer can be, I'd hate to have to answer to a crowd of them!

Anonymous said...

I don't immediately see the appeal of this new journal.

One thing that sets this new endeavor in stark contrast to _Philosophers' Imprint_ is that PI had some big names attached from the beginning, which I think significantly affected what it could do and how quickly it could be seen as a valuable publication.

Anonymous said...

anon 7:47 here. Thanks for the response, Z.

However, I'm not seeing the reviewer incentive. "Review what you want, when you want" is nice, but not actually an incentive. It's also true of reviewing at other journals; no one can force you to review what you don't want to review. That is, why should we think people will actually do this. Put another way, non-onerousness is not an incentive.

And the "no need for lengthy report" would seem to work as a disincentive for authors. It also runs contrary to the "quality feedback/decisions more quickly than with traditional journals" made above.

I worry, too, about cases of no one actually reading your piece. Is there some accountability for failure to act on papers?

Finally, "open-access publishing without a fee" already exists. Authors can retain rights to publish articles on their own websites / paper repositories (e.g., ssrn) and voila, free open access.

I hate to be a wet blanket here. I really do want some other model for publication. The journal model sucks for a variety of reasons. But I don't quite see it here. Hope I'm wrong.

Anonymous said...

On a related note (at least in the same category of using technology to pursue philosophical research), I see that experimental philosophers are now trying to elicit responses from professional philosophers to a series of new studies at this site:

They claim that, "The project hosts 17 different experimental philosophy studies designed by 29 philosophers, each working on illuminating a different philosophical question." And that they want to compare how philosophers and "the folk" respond to these studies.

I'd be interested in hearing (perhaps in a separate blog post) what the smokers think of this kind of project--or of experimental philosophy in general. Can this sort of project reveal anything interesting about philosophical topics? Or philosophy as a discipline?

Anonymous said...

Is the idea that I submit a paper, and then wait a month, and then *however* many people have looked at it, if more recommend it than not, it gets accepted? Or is the idea that once, say, 5 people look it, that will be enough?

The former model is bad for various reasons -- maybe 2 people 'accepted' it because they thought it was OK but didn't give it a careful read, thinking others would. The latter model may also have problems -- maybe if we've got to wait for 5 people to read it and vote, things will take just as long as they have been.

Anonymous said...

I am one of the philosophers who has already volunteered to referee. In my view the current system doesn't work very well, and alternative approaches are worth trying.

I doubt that it will be that difficult to convince senior philosophers of the merits of this proposal. If anything, they are an obvious constituency to get behind it, as they know better than anyone the drawbacks of the current system. Consider for example this remark by Colin McGinn's in an old thread on Leiter:

"Younger readers may be interested in the perspective of someone who has been kicking around for a while. I used to submit articles to journals in the seventies and eighties, but have given up in the last fifteen or twenty years, out of frustration with long delays and poor referee's reports. The last two papers I submitted (I'm excluding invited contributions) were "Another Look at Color" and "Can We solve the Mind-Body Problem?". The former was rejected by Mind and Philosophical Review and then accepted by the Journal of Philosophy. The latter was rejected by the Journal of Philosophy (without any comments) and then accepted by Mind (under Simon Blackburn). One result of this is that I have at least a dozen unpublished papers that I don't know what to do with and which might be of interest to the philosophical community. This is surely an undesirable state of affairs, but it is one that is not unique to me. I know many senior philosophers who are reluctant to embroil themselves in the journal submission process."

Posted by: Colin McGinn | December 03, 2007 at 09:56 AM

McGinn's remark about senior philosopher's being reluctant to go through the current submission process rings true. I can think of numerous big names in my area who I hardly ever see in normal journals any more, and who instead seem to publish primarily in various invited fora.

Perhaps Andrew C might drop a note to McGinn, asking if he and the "many senior philosophers" he refers to might be willing to submit a paper.

Anonymous said...

Colin McGinn's remarks show why famous philosophers don't need to give a shit. Can't get past blind-review? No problem, just wait for invitations. Or publish a book.

Anonymous said...

Anon 11:37 here. Anyone know the answer to my question above?:

Is the idea that I submit a paper, and then wait a month, and then *however* many people have looked at it, if more recommend it than not, it gets accepted? Or is the idea that once, say, 5 people look it, that will be enough?

The former model is bad for various reasons -- maybe 2 people 'accepted' it because they thought it was OK but didn't give it a careful read, thinking others would. The latter model may also have problems -- maybe if we've got to wait for 5 people to read it and vote, things will take just as long as they have been.


Is there a minimum # of votes needed for acceptance? If after 1 month, I've gotten above 50 percent of 'yes' votes, is the paper automatically accepted? (It'd be unfortunate, say, if I was going strong for a month or so, or two months, and then in the third month things go south, when perhaps other papers weren't held out for review for as long.)

There are various ways of going here, of course. Is there a policy in place yet?

Anonymous said...

I know this is a bit off-topic but it's sort of relevant and maybe there can be a thread about it. Anyway, here's my worry. Like many readers of this blog, I'm a young untenured faculty member. I think I have some really good papers, but thanks to the awful review process (and my tendency to work in isolation) it's been difficult to get many of them published. I've had a few minor successes but spend a lot of time waiting to hear back on several papers (and racking up rejections!)...all this while I'm scared to death of getting my best ideas "scooped" by someone else (i.e. published before me). Here, then, are my questions: how does everyone feel about posting working papers to places like SSRN? Is it a bad idea, because it "unblinds" what is supposed to be blind review? Or, is it a good idea, because it's a good way to publicly stake your claim to an idea ("getting there first"), even though it's just a repository for working papers? I just don't know, and I'm stressing myself out. The publishing game just sucks, and I could basically just use some advice on how to best deal with the issues I raised. Any ideas are greatly appreciated!

Andrew Cullison said...


Thanks to everyone for all of this great feedback. I wanted to give everyone an update.

1. We are at 120+ referees volunteers

2. We have an advisory board of 8 very well known philosophers, and we are currently in the process of building the board up more

3. We have a document that clearly outlines how the entire procedure will work. This addresses many of the questions that came up in this thread. We've shared the document with several philosophers for feedback, and have been refining it in light of comments. We'll post it soon for more feedback.

4. We're in conversations with a major research institution to be the archiver of all the content. We'll post more details about that soon.

Thanks again for all of your helpful feedback.