The Absurdity of Peer Review
I was reading my umpteenth news story about Covid-19 science, a story about the latest research into how to make indoor spaces safe from infection, about whether cleaning surfaces or changing the air was more important. And it was bothering me. Not because it was dull (which, of course, it was: there are precious few ways to make air filtration and air pumps edge-of-the-seat stuff). But because of the way it treated the science.
You see, much of the research it reported was in the form of pre-prints, papers shared by researchers on the internet before they are submitted to a scientific journal. And every mention of one of these pre-prints was immediately followed by the disclaimer that it had not yet been peer reviewed. As though to convey to the reader that the research therein, the research plastered all over the story, was somehow of less worth, less value, less meaning than the research in a published paper, a paper that had passed peer review.
Imagine reading about the discovery of the structure of DNA with that same reticence we use today: “In a recent Letter to the journal Nature, Cambridge University scientists James Watson and Francis Crick proposed a new structure for DNA (not yet peer reviewed). They claim their “double helix” model, a spiral of two strands of bases, both explains decades of experimental work, and provides a clear mechanism for copying genes. Their proposal drew heavily on data contained in Letters in the same issue of Nature from the teams of Rosalind Franklin (not yet peer reviewed) and Maurice Wilkins (not yet peer reviewed).”
Or consider this modern take on a certain scientist’s annus mirabilis:
“The past year of 1905 has been a remarkable for one Herr Einstein, who proposed no less than four theories new to modern physics in a series of papers. His first was on a much-anticipated explanation of the photoelectric effect (not yet peer reviewed), the second on how Brownian motion arises from the collision of invisible particles (not yet peer reviewed), the third on the equivalence between mass and energy (not yet peer reviewed), and the final paper updates Newton’s mechanics to be more accurate for objects moving close to the speed of light (not yet peer reviewed).”
These imagined reports are both eye-wateringly ridiculous.
The papers they describe are works that have formed the foundations of biology and physics as we know them. None of the classic papers on DNA, nor Einstein’s four in his miracle year, were peer reviewed. Indeed, only one of Einstein’s 300 or so published papers was ever peer-reviewed, which so disgusted him that he never submitted a paper to that journal again. It did not matter that the DNA papers and Einstein’s papers were not peer-reviewed. Nor did it matter for the many thousands of the classic papers published in the past century that were not peer reviewed: the work in them is the foundations on which science stands today.
Why then does peer review now apparently matter so badly that reports of research have to point out when it has not yet happened?
To me, that story on research into making indoor spaces safe during Covid-19 is a beautiful example of how modern science has got itself into a bit of a muddle over peer review: Science worships peer review, but it doesn’t know why. For as I’m about to tell you, it doesn’t do the job we think it does, so reporting science differently because it hasn’t been peer reviewed is absurd. To show you that, we have to go back to the start.
Peer review has a misleadingly long history, conveniently dated to when the Royal Society started asking for anonymous reports in the 1830s on whether some papers should be published in its Philosophical Transactions. But the use of peer review was patchy at best until the 1970s, when it became the norm at all kinds of scientific journals. Indeed the journal Nature, for many the pinnacle venue for publishing science, didn’t start peer review until 1973. In the short span of time since the 1970s, peer review has now grown from an occasional things into a beast that is devouring the time and energy and careers of scientists.
For peer review is now the rite of passage for almost every published paper. Scientists write a manuscript detailing their findings and submit it to a journal, where an editor then asks some other scientists, typically two to four, to review the work. On the basis of these reviews the editor makes a decision, typically one of three: to accept the paper as it is, ask for revisions based on the reviews, or reject the paper. Rejected papers go to another journal, to start the process again; revisions can include extensive new experiments, new analyses of data, or both. Getting a paper from that start line of its first submission to the published version can take months, often it takes years. This slow, grinding process is oft-quoted as being the “gold standard,” as being essential to ensuring only quality science is published.
Pandemics don’t conveniently hang around waiting for that slow, grinding peer review process to judge science. Science in the time of Covid-19 has had to be nimble, quick on its feet, has had to show its findings to the world without the layers of review-and-revise. We’ve needed rapid research into models of transmission, into immunity and reinfection, into public health messaging and effective interventions, into drugs to treat symptoms, machines to support the severely ill, and the development of radically new types of vaccine. So pre-prints, those manuscripts put on the internet for all to read before peer review, became the weapon of choice. Long common in physics, pre-prints in biology and especially medicine exploded in number during the pandemic.
And the media have dealt with this explosion by consistently pointing out when research has not yet been peer reviewed. Presumably they do this to warn the reader that the research lacks the safeguards that peer review brings. The problem with that warning is peer review guards against nothing.
Does it catch fraud or manipulations of data? No, patently not: peer reviewers are not omniscient, so they cannot divine made-up data, nor can they check all the outputs of a lab to see when they’ve simply copy-and-pasted data between papers. If they were, we wouldn’t have the website PubPeer stuffed to the gills with people flagging potentially serious misdemeanors in published papers, nor Retraction Watch’s endless reporting of papers so dodgy they’re expunged from the literature.
Does it prevent people from making basic errors of statistics or in how they designed their study? No, patently not: because otherwise there would be no “replication crisis’’ in many areas of science, where the claimed results of published work cannot be replicated.
Does it stop a plainly wrong or plainly nonsense paper from being published? No. There will always be somewhere to publish a paper, so the mere act of publication is no guide to the quality of science. Not even in the most elite journals.
Does it consistently chose the papers worth publishing? No. Rating consistency between reviewers of journal papers, conference papers, and grants is notoriously, offensively low. Which means that two groups of reviewers, given the same set of things to review, will agree on almost nothing. And none of these failures should be surprising: anyone with a basic grasp of statistics knows that a sample of, at most, four opinions is a terrible basis for any decision. Yet this is exactly what we do in peer review.
It is now the entrenched norm, yet we cannot make a scientific case for why we use it: we do not know whether the science we produce is better with or without it, but we do know at lot about what it can’t do. Even the positive benefits of peer review are limited, for while a paper undoubtedly can be improved by constructive reviews, much peer review is aggressive, rude, lazy, or just plain bad. And, good or bad, rarely does peer review meaningfully change the text or even the central claim of a paper. For after all, a paper’s authors typically know what they want to say, so they wrote that paper to say it: peer review is unlikely to persuade them to say something else.
Why then do we still have peer review? I’d wager that it’s a combination of simple inertia, and because journals rely on it for gate-keeping. When it did appear, peer review’s job was to help out editors who received papers beyond their expertise, and wanted some advice on whether to publish or not. Now it has become all about who gets chosen to be published in the most selective journals, who can convince a handful of reviewers their paper is worthy of publishing in that journal, and so dramatically increase the chances of their paper being read: for the primary purpose of the selective journal is to say “hey, read our stuff, it’s been hand-picked just for you.” It is a way for editors to filter papers based on the personal biases of two to four individuals. Without that gate-keeping role, it seems we would have no need of peer review.
This then is why I was so bothered about how Covid-19 research is reported: peer review is no guard, is no gold standard, has little role beyond gate-keeping. It is noisy, biased, fickle. So pointing out that some piece of research has not been peer reviewed is meaningless: peer review has played no role in deciding what research was meaningful in the deep history of science; and played little role in deciding what research was meaningful in the ongoing story of Covid-19. The mere fact that news stories were written about the research decided it was meaningful: because it needed to be done. Viral genomes needed sequencing; vaccines needed developing; epidemiological models needed simulating. The reporting of Covid-19 research has shown us just how badly peer review needs peer reviewing. But, hey, you’ll have to take my word for it because, sorry, this essay is (not yet peer reviewed).
Mark Humphries researches computational neuroscience at the University of Nottingham, U.K., and is the author of “The Spike: An Epic Journey Through the Brain in 2.1 Seconds” (Princeton University Press) out now.