Some thoughts on open and community sourced peer review

We have revised our paper on arXiv detailing some work on colocalisation analysis, a method to determine whether two traits share a common causal variant.

I have been following the debate about open peer review: not just reviewers for traditional journals signing their reviews, but the idea of community-sourced peer review: you publish your paper when you are happy with it, other scientists comment and point out weaknesses, you revise it appropriately and publicly. This all sounds like a great idea, and is part of why this paper was first published on arXiv. I really care about the appropriate use of statistical methods for colocalisation and think the topic of data integration is an important one. Having the paper on arXiv has been useful for sharing my paper with others, and for giving a reference url in talks. But, although some people have told me they read it, and I know some are using the software, no one has given me any criticism of the paper itself.

In parallel, I had submitted it to a traditional journal. The model there is that reviewers get asked by editors to spend some time reading a manuscript and comment in detail. This system has been subject to plenty of valid criticism for the delays introduced, and the often contrary reviewer 3, but the comments from one of the reviewers for this paper were fantastic: detailed, critical about areas where I had not been clear, and suggesting some important additional things to explore. I know, often, reviewers’ comments can seem pedantic, but occasionally you find a reviewer like this, who takes the time to read carefully everything you write, and spots holes and ways to improve it.

I don’t know who the reviewer is. I suspect s/he is a statistician working in or familiar with genetics, but perhaps not in the specific area tackled by my paper. I don’t know whether, under the community-sourced open peer review model, someone like this would have even read my paper. As a reviewer, I have to base my decision to review a paper on the fact that the editor thought I would know something about it and my reading of the abstract. Sometimes, the abstract doesn’t match the paper that well, and I realise that it will require two weeks of wading through treacle, or doing some unanticipated background reading in order for me to be able to give a fair and helpful review. Under a community sourced model, I would be able to see the paper in its entirety and would probably say no in some cases where I would have accepted based on the abstract alone.

There are some good arguments here about how reviewers could be motivated to submit reviews in an open peer review system, but I worry that it may cause us all (as reviewers) to focus on a smaller area of science: that most closely related to our work and which we feel most able to make comment on. The reviewer who is less intimately involved and can give an outside perscpective is often very useful. I don’t fully understand how the community-sourced peer review model will work. I like it as an ideal, but I worry a little about the practical reality of its implementation.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s