The Berkeley Earth Project has announced that they have found global temperature records are on the rise, in agreement with findings seen by groups from NASA and the UK’s Met Office/UEA. Normally such findings would work their way through the peer review system and get published in a reputable journal. Not this time…

In a highly unusual move, the results of the study, written up as a set of four papers, have been released on the project’s web site before the peer review process has been completed. If you are so inclined, you can access the raw data and run your own analyses. You can check exactly what has been done and verify for yourself the claims of the scientists. According to their site they are inviting “additional scrutiny as part of the peer review process.” According to various media reports, the lead scientist of the project, Richard Muller, believes that this free circulation of data prior to publication marks a return to the way science should be done. He told the BBC, for example, that:

That is the way I practised science for decades; it was the way everyone practiced it until some magazines – particularly Science and Nature – forbade it.¬†That was not a good change…

Wow! That is a significant statement to make for two reasons. It raises questions about something called the ‘Ingelfinger rule‘, where journals refuse to publish work that has been previously published. It also raises some significant concerns about results being reviewed by only a select group of a few (3-5) ‘peer’ scientists.

In this case they have effectively said we’ll take the risk on the paper(s) being rejected (because they’ve been substantially ‘published’ already), because we think our work has enough merit, interest and impact to be published by the journal anyway, publicised the work widely, got mainstream media attention and then asked for crowd review on top of peer review essentially saying that peer review is not enough scrutiny.

Let’s look at the Ingelfinger rule first. Many scientific journals insist that the results of studies submitted to them have not been published elsewhere previously. On the face of it the rule makes sense as it avoids duplicate papers in the scientific record and under its original meaning also protects the journal from publishing results that are not new. In practice, the rule is often criticised because it can lead to considerable delays in getting work published. In this case, the scientists have submitted the papers to Geophysical Research Letters who have such a policy. By releasing the papers and data whilst they are under review at a journal, they have presumably blown the rule out of the water with the risk that the work will not get published. However, this is significant work. Will the journal reject it on a technicality, now that the content is accessible, in the public domain and had plenty of mainstream media coverage? It will be interesting to see what happens.

Next, let’s look at the peer review versus crowd review issue. By releasing their work early, they have asked for much more feedback or review than they would receive normally under peer-review. Whether or not they will get it is another thing; we’ll just have to wait and see. However, by going public with the papers and data they have singularly answered some of their biggest potential critics – the deniers and sceptics – before they have had the chance to open their mouths. Previous studies in this area have been criticised because of a lack of transparency on methods and collection of poor quality data. There are also (often unfounded) accusations that peer review can be a bit of a inside job in that only one side of the argument ever gets through (trust me, peer review is anything but). In one sweep, the Berkeley study looks to have answered these issues by going public early. Anyone can inspect their work and anyone can review their work.

Ok, but this raises a considerable question. If review by a handful of peer scientists was deemed by the authors to not represent enough scrutiny, what does that mean for biomedical sciences or for that matter the whole of science? What about all those big questions of our time? Is peer review as it is now good enough for say, vaccine development? What about antibiotic resistance? What about topics like genetic modification of foods or embryonic stem cell treatments?

Opening data sets up for world-wide scrutiny seems to have been deemed necessary in this case for answering potential critics before they had a chance to open their mouths. This approach might well become required in other areas too. Perhaps the temperature is about to be turned up on peer review as we know it.

Tagged with:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>