Turning peer review upside down

We recently hosted Mark Patterson, Executive Director of eLife, as part of the Science & Society Program at EMBL Rome. In case you’re not familiar with eLife, it’s a journal supported by four major Life Science research funding bodies: the Howard Hughes Medical Institute, Wellcome, the Max-Planck Society, and the Wallenberg Foundation, with a mission to explore a more responsible selective peer review system. It’s been a surprising success. Although eLife follows DORA guidelines and formally asked not to be listed by Thompson Reuters, the ascribed Impact Factor is already 7.9. It is widely regarded as a top journal in the Life Sciences on par with the Nature specialty series.

Mark Patterson is Executive Director of eLife

As promised, eLife has been experimenting, and the highlight of Mark’s talk were the results of a recent trial in which they turned peer review upside down (see this paper from Bodo Stern and Erin O’Shea for the origins of the trial). For a limited time, manuscripts sent to eLife were accepted BEFORE being sent out for review. Sounds crazy – I know. No need to respond to the reviews if you don’t like them – your paper is already accepted! The hitch, however…

…is that the reviews and how you decide (or not) to respond are published alongside your paper. Shame is the most powerful emotion, they say, and the system surely forces you to be pretty sure you’re ready to stand behind your work before submitting. In that way, it’s a bit like accumulating comments and retorts on your latest BioRxiv publication – except, of course, that the eLife editors have to like your manuscript first.

The trial is closed now and it’s not clear they will switch to the accept-then-review system any time soon. It was associated with some distortions. For example, a larger fraction were recalled following review –apparently under fear that the reviews would be published, or perhaps just having gotten some good advice. It was associated with an increased rejection rate at the editorial level (from 72% to 78%), and it appeared to show some bias against early career researchers. But I think the trial shows that there is much to be discovered and less to fear in radical peer review publishing alternatives. The aim was to break the power relationship between authors and reviewers and, in this, it clearly succeeded. Keep tuned to the eLife Twitter site for the latest developments and don’t forget to follow EMBL Rome’s very own ex-blogger Emmy Tsang, eLife’s new Innovation Community Manager.

Author: Cornelius Gross

Neurobiologist interested in the molecular and cellular mechanisms of the control of instinctive behavior associated with defense, reproduction, and ingestion; EMBL Group Leader since 2003, Senior Scientist & Deputy Head of Unit since 2009, postdoc with Rene Hen at Columbia University, PhD with William McGinnis at Yale University 1995

One thought on “Turning peer review upside down”

  1. The way how research is performed and how knowledge is shared is radically shifting towards open systems in the last decade. Nonetheless, when it comes to valuing science and scientists, the question ‘what have you done?’ has been substituted by ‘In which journal did you publish?’ and ‘what impact factor?’. As a fresh first year PhD student, it is being quite tough for me to realize that this misleading way of valuing science makes publishing slow and painful, and promotes scientific misconduct. It seems to not to be so important anymore to do advanced, reproducible and ethical science. Nowadays, it is a fact that the highest reward a scientist can have is a publication in Nature.
    In my view, eLife’s trial is a powerful initiative to start to make some changes in the publishing system, but there is a lot of work to be done. Mark envisions the future role of journals as entities that gather together boards of well respected editors and reviewers that will curate science to ensure scientific quality, in a transparent and open way. Mark’s talk gave food for thoughts in our institute in which most group leaders are early career researchers, and that might make us reconsider the role of peer review.
    Some of the most influential texts in the history of science were never peer reviewed, such as Newton’s Principia Mathematica, Einstein’s paper on relativity and Watson and Cricks Nature paper on the structure of DNA. Instead, in recent years, papers published in high-profile journals have passed peer review and then heavily criticized by the scientific community, and even retracted (check http://retractionwatch.com/). This makes me question if peer review is positive at all for the scientific community. Maybe science evaluation should be based on the acceptance of the scientific community. Luckily its very easy to give direct feedback to authors nowadays though biorxiv. By the way, when typing peer-review system in google, second suggestion is ‘broken’.

Leave a Reply

Your email address will not be published. Required fields are marked *