Improving peer review?

While the peer review process often draws criticism, most would agree that it is a necessary, indeed integral aspect of conducting research. Even the most ardent critics acknowledge that the simple, even elegant, system has shown itself to be at least an adequate tool for a difficult job.

But peer review is not perfect and when it is itself scrutinised, there are various shortcomings that one might wish to address. Now, a group of researchers based at the University of Jyväskyklä, in Finland, want to take peer review and make it better. As they see it, the problems with peer review are that the process is slow, that reviewers receive no credit or recognition for their work, and that the quality of reviews is often poor. They also note that peer review is often attacked with accusations of bias, and as rejected papers trickle down the hierarchy of journals, the review process is repeated, needlessly consuming the time of an ever-increasing number of reviewers.

Their solution is a new website – Peerage of Science (PoS) – described by its founders as ‘a social network for peer review’. ‘This is not a revolution,’ says one of the founders, Janne-Tuomas Seppänen, ‘but a re-volition’. For Seppänen, PoS is the solution to the shared problems of peer review, that will be driven by the scientific community’s desire to change the system and to progress and safeguard their science.

— Peerage of Science’s founders: Janne-Tuomas Seppänen, Mikko Mönkkönen and Janne Kotiaho

At its heart, PoS is a simple idea. It is a web tool that facilitates peer review – authors upload their manuscripts and the community reviews them, following an automated process controlled by PoS. So far, so similar, but it is in the details of the system’s operation that PoS hopes to tackle some of peer review’s problems.

Perhaps most interesting of these is the mechanism for review. Having established their credentials (a publication in a peer-reviewed journal will suffice), participating reviewers choose whichever manuscripts take their fancy, with deadlines (set by the authors) that fit their schedule. Furthermore, all the reviews (called peer essays) a manuscript receives are themselves then subject to review – each reviewer is required to score the other reviews of the same manuscript. In this way, reviewers receive a rating (called the peer essay quality) based upon their peers’ opinion of their reviews, designed to give reviewers some recognition for their contributions and to encourage a higher quality of review. In a final twist, every review a manuscript receives generates two review obligations for the authors. So you don’t get something for nothing: to use the system, you must actively support the system.

To address accusations of bias that often dog the traditional peer review approach, the whole thing is conducted anonymously. Although reviewers are permitted to reveal themselves at any point and indeed, if they wish, they can even publish their ‘peer essays’ in Proceedings of the Peerage of Science. That’s right: a journal of referee reports. Inscrutable as this might appear, the idea is to give reviewers another means to demonstrate their expertise, to further reward their time and effort.

But of course the aim of peer review is to get published, so how does PoS include that essential third party – the journal editor? Ultimately, the idea is that journals will participate in the PoS system – journal editors can track relevant manuscripts as they proceed through the PoS system and can offer to publish them at any point during the process. At the moment, however, this will only happen if ecology is your thing (Ecography is the only journal currently participating). But the system also allows an author to export the PoS review to support submission to any journal. And this is an essential aspect of PoS – it is in providing the outcome of PoS reviews to publishers that the founders hope to commercialise their efforts.

The PoS team have clearly got big plans for their system and are passionate about improving peer review. And looking through the current peer community, they are not alone – various august institutions are already represented in PoS’s peer community (though, at present, nary a chemist among them). But while the aspirations of PoS have to be commended, there are some important questions to be asked. Will publishers be willing to recognise this system of peer review? And is it truly any less susceptible to abuse than the current review methods; does the online anonymity even enable it? And what if nobody reviews your paper? Mike Foster makes some very good points in his blog and it’s clear that the debate is just getting started (and PoS are eager to be involved).

It would seem that the success of this enterprise relies most heavily on the peer community. As with any social network, achieving a critical population level is the key. So are scientists willing to participate, or would they rather just let the publishing houses continue take care of the whole process on their behalf? Perhaps the greatest challenge facing PoS is simply that the established publication route is so familiar and works well enough that the improvements offered by PoS won’t be attractive enough to overcome the inertia of the status quo. What do you think? Would you join the peerage of science? Is this the future of peer review?

Philip Robinson