by Wei Li

News about exciting and novel science is being published every day, whether it’s about a new technique for rejuvenating skin cells, or a new breakthrough in fighting drug resistance for cancer patients. However, when you read these news articles, how can you assess if they are trustworthy? How do you know if a new technique actually works or if the data used to make a conclusion is analyzed correctly?

One way to know would be to read the actual scientific article to check the data, but a layperson, or even a fellow scientist in a different field, may not be able to properly evaluate the data presented and comment on the validity of the conclusions. That is where peer-reviewed journals come into play. 

Peer-reviewed journals contain articles that are not only conducted and written by experts, but reviewed by several other experts in the same field as well. This peer-review system is crucial for maintaining a level of rigor in scientific publications, as well as ensuring a level of trust in the scientific community by the general public. Unfortunately, however, the world of peer-reviewing has many issues that can potentially impact its credibility. To fully understand this potential, let’s first look at how the reviewing process works.

The peer-review process

Different peer-review journals have different reviewing processes, but they usually follow a similar structure. In general, when a manuscript is submitted to a journal, the journal editor sends the manuscript to at least two reviewers. These reviewers are typically experts in the field, but should have no direct affiliation with the authors. These experts then read and assess the article, providing feedback that the editor sends to the authors. The authors can then make changes, conduct more experiments, and improve upon the manuscript based on the reviewers suggestions. After the authors have answered all of the reviewers’ comments and concerns, the manuscript can either be accepted or rejected for publication by the journal (Figure 1). 

At the scientific journal Science, this entire process, from original submission of a manuscript to its final acceptance, takes about 123 days on average. If that seems like a long time, a quick search in the Review Speed Database will tell you that this is in fact a typical amount of time for a manuscript to be accepted—and this data is not accounting for the countless number of manuscripts that are rejected! The journey to being published in a peer-reviewed journal can be a long and arduous process for researchers.

A complicated process for everyone involved

The reviewing process can not only be difficult for the researchers who have to go through rounds of editing and answering reviewers’ comments, but taxing for the reviewers as well. Most of the time, reviewers are not paid for their time spent reviewing manuscripts, and are usually fellow academics who are expected to take time away from their regular job to be a reviewer. In fact, a study shows that a typical academic who works on reviews completes about 4.73 reviews per year. While this number may seem small, each review takes about four to five hours to complete; globally, the total time spent on peer reviews was over 100 million hours in 2020—equivalent to over 15 thousand years!

Academics are expected to be willing to dedicate this amount of time to review for altruistic purposes, as well as for other non-monetary rewards, such as obtaining free journal access, being acknowledged in journals for their efforts, the possibility of receiving favor from journals when they need to publish papers themselves, and many others.Furthermore, the journals themselves are also overwhelmed. There are a high number of papers submitted for review every day (an estimated 21 million articles were reviewed in 2020!); additionally, submission rates can fluctuate greatly over the course of a year, causing editors to be unusually swamped during certain times. 

Potential problems of peer review

Because of how overwhelming the review process can be, the results are not always consistent between different articles and journals. Particularly, the decisions of reviewers can be inconsistent. One study showed that recently published articles, when resubmitted a few months later, are often rejected by the same journal – most of the reviewers did not detect that it was a resubmission, and the articles were frequently rejected due to “methodological flaws,” showing the volatility of reviewer decisions. This may be due in part to the disparities in opinions between reviewers, making it very difficult to submit a paper that will be liked by all of the reviewers. In fact, another study did a probability analysis and showed that it was so unlikely and unpredictable to get two reviewers to agree, that getting a paper accepted by  both reviewers has a similar probability to throwing a dice.

Additionally, reviewers are of course humans too! They will sometimes miss critical information in a paper or have personal biases when reviewing, causing dubious research to sometimes be published. Furthermore, another study shows that there may be a bias in favor of the institutions that the reviewers themselves are affiliated with. After all this work, published, peer-reviewed works can still be retracted, with one of the most notable examples of this being from a few decades ago, in which a paper was published in the Lancet that linked autism to vaccines (Figure 2). This paper was later retracted for many reasons, including data manipulation, low sample size, conflicts of interest, and countless other pieces of evidence contradicting the claims. As you can see, not every paper that is peer-reviewed is a mistake-free paper with good science.

There is also a gender bias in selecting reviewers – despite a significant portion of researchers being women, women make up a much smaller fraction of reviewers. This survey observed that authors, regardless of gender, suggest mostly male peers as reviewers to their editors (Figure 3). Along these same lines, another paper determined that female reviewers are less likely to be chosen by peers than if a reviewer was randomly selected. This widespread gender bias may then lead to further biases in the review process. This same study showed that there are fewer female authors publishing than what is expected based on the population of female researchers, possibly due to a gender bias similar to the one present in reviewer selection.

What can be done to improve?

Overall, there is a lot of work being done to improve upon the peer-review system. For example, journals are now putting in more effort to retract incorrect papers, so that papers that slip through the cracks are taken down in a more timely fashion, working to prevent the spread of false information. There are also ongoing debates over whether journals should start paying reviewers a compensation fee for reviewing – perhaps this would solve the inconsistency in peer-reviewing by giving reviewers incentives to pay more attention and be more mindful about what to reject or accept. However, this could make the cost of publishing and reading journals even higher, potentially reducing the number of publications a journal can publish in the future.

With the internet, it is also becoming a lot easier to facilitate discussion of a scientific article,  allowing for additional review by the public beyond the traditional peer-review. For example, PubPeer is a site that allows scientists to review and talk about published research, while medRxiv and bioRxiv allow scientists to post their manuscript before peer review—also known as a preprint—so that the public gets a chance to read and review the papers more quickly. However, this rise of preprint publication, while useful for allowing scientists to garner feedback from the larger scientific community more quickly, is affecting the ways that news outlets are reporting scientific discoveries – many large media sites often report about preprint articles without mentioning that they are not peer-reviewed yet. Moving forward, news outlets need to realize the importance of reporting the preprint nature of these articles, and consumers need to be mindful of if the new research they are reading is indeed from a peer-reviewed journal.

Now, back to the original question: when reading an article about a new scientific discovery, how do we know if we can trust the data? While there are a lot of factors to consider, finding out if the article is peer-reviewed can be a quick litmus test for credibility. However, just because a paper is published in a “peer-reviewed journal,” does not mean that the paper is completely fact-checked, unbiased, or correct. The peer review system is not perfect, but for now, it is the closest thing we have to ensure academic rigor. Like everything in life, we just have to take these articles with a grain of salt.

---

Wei Li is a fourth year graduate student in the Chemistry and Chemical Biology Ph.D. program at Harvard University. She studies the chemistry of gut bacteria and their effects on host physiology.

Cover image by Pexels from pixabay

For more information:

Read more about the difficulties of peer review here, here, and here.