Peer review, despite its flaws, is one of the most important pillars of the scientific process. So preprint servers, which make scientific papers that have yet to be reviewed or published available online, have been slow to catch on in many fields.
But then came the pandemic.
“COVID changed everything,” says Jim Handman, executive director of the Science Media Centre of Canada. Scientists, science communicators, and journalists who had been wary of using preprints in the past suddenly felt the urgency to get important new information out as fast as possible to help deal with the unprecedented public health threat. The use of preprint servers skyrocketed. Now, everyone is adapting to this new way of working, developing best practices to harness the benefits of increased speed and wider reach while mitigating the risks of sharing unreviewed science.
Most of the time, the world of scholarly publishing moves at an almost glacial pace. New publications can take months or even years to wind their way through the process of peer review and publication. Even then, they can be hard to access for most people. So 30 years ago, some scientists started posting their work in online repositories before it had been formally reviewed and published. ArXiv, which shares research on math, physics, and astronomy, was the first to launch in 1991. It was followed by repositories for other subject areas over the next few decades.
“Preprints accelerate scientific communication, which is a great thing for science overall and for the authors of the paper.”
For most scientists the main selling point of preprints is the speed at which they can share the results of their research, and find new work uploaded by their colleagues. “Preprints accelerate scientific communication, which is a great thing for science overall and for the authors of the paper,” says Jessica Polka, executive director of ASAPbio, a non-profit group that bills itself as “working to promote innovation an transparency in the life sciences.” They can also be a useful way to get early feedback, find new collaborators, and establish who was first with a discovery, she says.
The need for speed
Devang Mehta, a postdoctoral researcher in plant biology at the University of Alberta, says for him speed is the biggest draw of preprint servers. “It gets your work out faster than traditional publishing routes,” he says. “Otherwise, it can be months or years before the community gets to see what you have discovered.”
Preprints have dramatically changed the way science works, he adds. Dr. Mehta posts all of his own work as preprints, and they are also how he keeps up to date with research in his field. “Most of the papers I read are preprints. When I do find a journal paper, I have usually already read it as a preprint.”
Concerns that posting a study as a preprint could lead to getting scooped, or having your paper rejected by a journal, are largely disappearing as well, says Dr. Mehta. Preprints are actually a good way of ensuring you get your name out first. ”If you’re stuck in review, you have no claim to priority,” he says. And most publishers are now comfortable with accepting papers that have previously been posted as preprints. Dr. Mehta serves on the early career advisory group of the biomedical journal eLife, which only accepts manuscripts that have been posted as preprints. More traditional publishers like Springer Nature even encourage the use of preprints and do not consider them to be prior publication (which can be a reason for rejecting submissions).
While younger scientists tend to be more comfortable using preprints to share their work than some of their older colleagues, Dr. Mehta says, everyone makes sure they are reading the latest preprints. “You can’t afford to miss out on your competitors’ work, or a useful new method.”
That attitude has taken longer to catch on in some scientific fields than others. While preprints have long been popular in physics, fields like medicine have been much slower to adopt the practice. That’s likely because the stakes are higher, should research with an impact on human health turn out to be unreliable after going through peer review, says Mr. Handman.
The beginning of the COVID-19 pandemic, however, changed things as the benefits of timely communication in most cases outweighed the potential risks of sharing unreviewed work. There was a dramatic rise in deposits to the medRxiv preprint server, which focuses on health sciences, in the early months of the pandemic. Up to 40 per cent of COVID-19 papers were first published as preprints, says Dr. Polka, of ASAPbio.
“It was the first time preprints were used at that scale in medicine,” she says. “We saw a lot of very important discoveries released first as preprints, for example early clinical descriptions, or the use of prone positioning for hospitalized patients.”
Juan Pablo Alperin, co-director of the Scholarly Communications Lab at Simon Fraser University, says the huge bump in preprint activity at the beginning of the pandemic also spilled over into non-COVID-related fields, and led to a broader acceptance of preprints more generally. But he says it is starting to taper off now that the urgency of the pandemic has faded, and it remains to be seen how much of that bump will persist over time.
Media impact
While researchers began turning to preprints in greater numbers during the pandemic, an even bigger change took place outside academia. Suddenly, journalists and members of the general public were coming into contact with preprints regularly, often for the first time, with little or no understanding of how they should interpret the research.
Before COVID-19, the Science Media Centre of Canada never included preprints in its weekly roundup of interesting papers for journalists, says Mr. Handman. “I can’t be an expert in everything, so the only way I have to validate the science is peer review,” he says. “So I felt it was too risky.”
But after COVID-19 hit, with important new discoveries coming out every day, Mr. Handman felt a sense of urgency he’d never seen in his career. “There was a pressing need to report on preprints, we couldn’t wait for peer review,” he says. “People needed this knowledge as fast as possible.”
With so much preliminary and unverified information being released, those whose job it is to communicate science to the public began taking extra care to ensure they are explaining the potential caveats as clearly as possible. Samantha Yammine, a science communicator based in Toronto, has long advocated for preprints as an important way to increase access to science.
Dr. Yammine treats preprints much the same way she does any publication, carefully evaluating the conclusions and the track record of the authors – after all, poor-quality studies can still find their way through peer review and get published in top journals – but she says preprints must be handled with extra care. “They can be easily misinterpreted, especially by people with less genuine intent.” So Dr. Yammine spends time checking what other experts in the field are saying about a new preprint before sharing it with her large following on social media. “I let people know that things might change, and to take it with a grain of salt, but if experts agree and there is an important take-away, that is urgent to get out,” she says.
Many journalists operate in the same way. Roxanne Khamsi, a freelance science journalist based in Montreal, says that as the number of preprints in health and medicine exploded during the pandemic “it changed the process of reporting for all of us.”
“Two years ago, there would be more of an internal dialogue in my mind about whether a preprint is really worthy of coverage,” she says. “Now I’m much more agnostic about whether something is published because so much more goes to preprint first, and the speed of news has accelerated.”
A bigger concern, says Ms. Khamsi, is the rise in “science by press release,” where results are released publicly without even going through a preprint, let alone a peer-reviewed journal. Much of the early data on vaccine efficacy was released this way by the manufacturers, for example. “I don’t stay awake at night worrying about preprints, but I do worry about press releases,” she says.
Like Dr. Yammine, Ms. Khamsi tends to treat preprints like any other paper, getting comments from other experts to contextualize the findings and taking a close look at things like sample size and control groups. “We’re not expert reviewers, but we should have these basic skills,” she says.
The context conundrum
While Ms. Khamsi doesn’t think the average reader knows much about the intricacies of peer review, she says it is still important for journalists to make the distinction that the science they are writing about is still in its early stages of publication and explain why people should be cautious in interpreting it. Mr. Handman has a different point of view. He says it can be a tricky concept to convey to the public, and may end up raising more questions in a reader’s mind than it answers.
Not all reporters provide this context for their audience. A study by Alice Fleerackers, a PhD student in Dr. Alperin’s lab at SFU, analyzed more than 450 news stories on COVID-19 from around the world that featured preprint research in early 2020. She found that only about half of them included any mention of the work’s preliminary status. Big, established publications like The New York Times and The Guardian were just as likely to fail to identify preprints as newer outlets like Medium or Yahoo! News, Ms. Fleerackers says. In a follow-up study (available now as a preprint) she found that while many journalists are aware of the issues surrounding preprints, and do risk-benefit calculations before using them in news coverage, they often aren’t sure how to verify or challenge their conclusions. “I’m not against covering preprints,” says Ms. Fleerackers. “It’s not a bad thing if it is done in a responsible way. They just have to think more deeply and critically about them.”
“It’s an opportunity to continue to educate the public on how science is conducted and generate awareness of the importance of peer review including all of its limitations and shortfalls.”
But the risks involved may not be as high as they may seem. Two papers published in PLOS Biology in February 2022 compared preprints with the versions that were later published in peer-reviewed journals, and they came to what some might think is a surprising conclusion. “Most don’t change much in the substance of their main conclusions when they are reviewed and published,” says Dr. Polka, who co-authored one of the papers.
Of course, those studies only account for preprints that eventually made it into a reputable publication. Dr. Alperin points out that studies touting hydroxychloroquine or ivermectin as wonder drugs to treat COVID-19 were shared widely as preprints before being largely debunked. “A single study can have a huge impact,” he says. “It’s hard to capture the harm they can do.”
But reporting on preprints can also have an impact beyond just getting important information about new science out there. It offers a way to report on the process of science and explain how it works, says Dr. Alperin. “It’s an opportunity to continue to educate the public on how science is conducted and generate awareness of the importance of peer review,” he says, “including all of its limitations and shortfalls.”
And highlighting those flaws can, paradoxically, help improve the public’s opinion of the whole endeavor, Ms. Fleerackers points out. “Studies have shown that framing science as a process that includes failure can actually increase trust in science,” she says.
This article is superficial, flawed, and incomplete, and amounts to a puff piece. In fact, two preprints from Canada caused massive problems — one that incorrectly claimed cardiac complications from vaccination from a group at the University of Ottawa, which was later retracted but not before the right-wing press absorbed it into canon, and another vastly underestimated the protection of vaccines, and it too was later retracted. Both errors would have been caught in peer review. Neither claim, when made correctly, mattered. Only the errors made them sensational, and those led to public health problems.
As for the PhD-Dr. Polka’s paper in PLOS One, it has major flaws, including selection of preprints (they are often posted on multiple servers, but they chose one), data excluded from the initial analysis, and the MAJOR fact that most preprints are posted AFTER peer review, making a comparison between preprint and paper more like a comparison around copyediting revisions.
Reporting like this by non-experts is why we need expert intermediaries at the front-end of science. Journalists are not equipped — no matter what they may think — to understand complex statistics, highly nuanced and specialized scientific findings, and so forth. They also have different incentives and deadlines.
I could go on about the problems caused by preprints (ivermectin ring any bells), and the public health problems — including generalized confusion and exhaustion — they have caused.
It is also equally likely that preprints may confound scientific progress by confusing the scientific record, making it unclear what a “claim” actually is, and increasing the burden on researchers throughout the data-gathering, report-writing, and publication processes.
Please do better next time. There are people tracking this carefully whose expertise you could have accessed if your goal hadn’t been a puff piece.
I find it odd, Kent, that you are touting the value of peer review over preprints while simultaneously discrediting the peer reviewed work of Dr. Polka et al.
PLOS One is what I categorize as a “commodity journal,” with no clear affiliated expert community. That made it lower its review standards, and makes its reviewers often unqualified. There is no clear community of experts to draw from.
More broadly, if peer review can fail, why would you support no organized and consistent review?