In February, the journal Nature and its sister publications announced a new policy for their peer review process (the evaluation of submitted articles by experts in the field). The journal normally operates on the basis of single-blind peer review–anonymous reviewers see the authors’ names and affiliations. The new policy will allow authors the option to remain anonymous to the experts reviewing their work. Will this change address growing concerns about the reliability usefulness of peer review in modern scientific work?

Figure 1 ~ An outline of the peer review process. Each steps can take several days or several months to complete.

“Reviewer #1: the conclusions are supported by the results and discussion.
 Reviewer #2: The main inconvenience of the paper is that the conclusions are not supported by the experimental results. “

– Mar 12th, 2015 submission to shitmyreviewerssay.tumblr.com, “a celebration of the harsh, the weird, the passive aggressive, the active aggressive and the downright mean comments practicing scientists receive on a regular basis.”

Peer review is integral to the legitimacy and authority of modern science. Before deciding whether to publish a scientific manuscript in a journal, an editor asks several scientists who have relevant expertise to comment on the quality, relevance, importance, and novelty of the work. This step serves as basic quality control, ensuring that there is a correspondence between the authors’ claims and the data, and that the experiments are novel and appropriate for the journal. The underlying motivation for this process is to facilitate the development and implementation of the best medical and scientific advancements—progress [1, 2].

Reviewers often provide insight that improves papers before publication. However, the peer review system used by many scientific journals is susceptible to the influence of personal or professional biases. Known as single-blind peer review, this system allows the reviewers to comment anonymously on manuscripts that reveal the author’s name and institutional affiliations [3]

In the context of growing concerns about quality and effectiveness of peer review, Nature Publishing Group announced in February that its monthly research journals and the highly prestigious weekly Nature will offer the option of double-blind peer review, in which a manuscript’s author is anonymous to the reviewers [3]. The announcement has reinvigorated an ongoing, unresolving conversation about the role and responsibility of peer review to the scientific and lay communities.  A closer look reveals deeper cracks beyond the issue of bias.

A Bit of Historical Context

The peer review system is often compared to the democratic political system—one that is far from perfect, but the best we have [2]. Indeed the comparison between a political democracy and the community of scientific peers is not accidental, as the two systems formed alongside one another. An example of their close ties is a famous feud between the political philosopher Thomas Hobbes and the natural philosopher Robert Boyle in 17^th century England over the existence of the vacuum. Robert Boyle laid the foundation for the scientific method while designing elaborate experiments to demonstrate the existence of a vacuum. He invited gentry peers (peers in the social sense more than in the sense of fellow scientific expert) into his laboratory as witnesses and published detailed accounts of his methods for peers further afield in order to convert his individual observations into matters of established knowledge in his community. This reliance on the individual senses and opinions worried Thomas Hobbes, because emphasis on individual opinion could undermine the strong government he thought was necessary to maintain social order. These political worries informed his objections to the possibility of a physical vacuum, just as much as Boyle’s own understanding of the social order informed his search for the evidence in a vacuum’s favor [4].

Boyle’s empirical approach to knowledge-making evolved into the modern scientific method. Scientists continue to rely on the credibility of their peers (fellow expert scientists) to take individual observations and enter them into a body of knowledge—now, the process of pre-publication peer review. This process did not actually become standard in scientific publishing until the mid 20^th century. Scientific journals have existed since Boyle’s time, but lack of deep specialization allowed editors to judge the validity and importance of most submitted manuscripts on their own or through informal consultation with scientist colleagues. The institutionalization of peer review flowed from of the explosion of specialization in research in the second half of the 20th century. It allowed editors and publishers—themselves scientists, now deeply specialized—to obtain the opinion of experts to help them judge the worthiness, and filter through the sheer volume, of ever-more niche experimental ventures [5, 6].

 Peer Review: The Basics

The reporting of scientific findings now follows a standard process. When a scientist has made a finding that she would like to report to the scientific community, she writes an article detailing her experimental rationale, approach, data, and conclusions, chooses an appropriate journal and submits the work to an editor of that journal. If the editor is interested in the content of the article, she can then send it to colleagues (other scientists—leaders of research groups or experienced researchers) who have experience with the type of work or question that the author is trying to address. These colleagues (scientific peers of the author) are asked to comment on and review the correspondence between the author’s claims and presented data, and suggest improvements or point out oversights, alternative explanations, flaws in experimental design or logic, etc. The editor can then use these comments to decide whether to publish the paper, ask for modifications, or reject for publication in that journal (for an illustration of this process, see the figure).

Importantly, the purpose of peer review is not to decide whether a given finding is “true” or to make it incontrovertible “fact,” nor necessarily to detect fraud or plagiarism (problematic developments in an increasingly competitive funding environment). Its main purpose is to assess the novelty, importance, and scientific cogency of the presented work—to ensure that it is worth the attention of the scientific community.

The Trouble with Single-Blind Peer Review

 “I have three main objections to this paper: it is self-contradictory, it is functionally obsolete, and it has been submitted to the wrong kind of journal.” –March 15, 2015. shitmyreviewerssay.tumblr.com

In an ideal community of peer scientists, peer review would force scientific work to pass a uniformly high bar based solely on its merit. Scientists, like all people, have trouble putting their biases aside. Single-blind peer review allows conscious and unconscious biases regarding the author’s professional reputation, gender, race, or institutional affiliation to influence reviews. Concealing the identity of the author would, in theory, remove these biases. Yet the high degree of specialization in modern science makes the preservation of anonymity difficult. The small size of some fields, pre-publication presentation of data at professional conferences, and the citation of authors’ own work (a common practice) can allow reviewers to guess the author’s identity[1,7]. Although the effects of such a guessing game on reviewers’ attitudes have not been studied, these concerns nevertheless contribute to broad ambivalence towards double-blind peer review in science (humanities and social science journals generally prefer double-blind peer review)[8].

Nature Publishing Group’s attempt to introduce double-blind peer review to high stakes research is an important step towards resolving these issues. The trouble with this particular implementation of double-blind peer review, however, is that it is optional. Why would a well-respected scientist forgo the subtle “bump” of reputation by choosing anonymity? Only those authors who have reason to fear bias would choose the anonymous option, perhaps cuing the reviewer to their reputationally disadvantaged position! Indeed, two Nature subsidiary journals that have had this policy in place since 2013 have found, despite popular approval of the double-blind approach among surveyed scientists [7], only a quarter of submitting authors have opted for anonymity. If Nature Publishing Group hopes to test whether author anonymity improves the quality of peer review or influences manuscript acceptance, the optional implementation is no way to go about it. Another problem with this implementation is that the editors still know the author’s identity, and can still act with bias. At a high-profile journal like Nature, the editors not only make the final decision about whether or not to publish the paper, but serve as the initial filter (that means immediately rejecting the large majority of submissions without peer review) [9].

 Beyond Bias in Peer Review

Other schemas for peer review have been proposed, but none is a clear champion for fairness. Fully open review, where both the authors’ and reviewer’s identities (and maybe comments) are made public, may keep everybody honest. However, there is still the complication of maintaining working relationships and reputations. These can prevent reviewers from being appropriately critical, or discourage young scientists from contributing their opinion [8]. Others have suggested that in the digital age, pre-publication peer review is hardly necessary—let post-publication on-line discussion sort things out (but beware those online trolls!) [9]. A biomedical extension to the online preprint repository arXiv, which chiefly serves the physical science community and operates without official peer review, may facilitate this transition in the future. This approach may enhance innovation, helping give voice to those scientists whose unorthodox ideas may have trouble getting through the peer-review filter of their conservative colleagues [1,9].

In fact, other journals have made similarly half-hearted attempts at offering alternatives to single-blind peer review (including Nature in 2006 with open peer review), but the results have been consistently inconclusive and unsatisfying, with low participation [1,3]. The lack of enthusiastic uptake points to deeper issues beyond the question of bias, fairness, and anonymity. The bewildering inconsistencies between paired reviewers (see the introductory quote), for example, can leave the authors feeling that the fate of their work relies on chance and whim. Furthermore, while 86% of scientists enjoy serving as reviewers [7], many also find the process inefficient and expensive, cutting into valuable time in the lab [1, 7]. Finally, all forms of peer review have potential for abuse. For example, an unscrupulous reviewer may borrow ideas or deliberately tank or hold up the editorial decision about a competitor’s paper. Following a trend of scandalous retractions, just last week, the open-access publisher Biomed Central retracted dozens of articles with fake or compromised peer-reviews [10]. Common to these problems is the absence of standardization and transparency in peer review. Reviewers receive general guidelines but little specific training, feedback, or material reward—they learn on the job, from the reviews they receive on their own submissions [1,7] Addressing this systematic deficit may indeed go further to improve peer review than changing the rules of author or reviewer anonymity.

In asking how peer review can be improved, it is important to remember that it serves both individual scientists and the taxpayers and private funders who invest in research. And in trying to understand the faults of the currently dominant schema of peer review, it is useful to take a broader view. In the real world, experiments face logistical challenges, funding is highly competitive, and career scientists must “publish or perish.” Peer review should be studied and improved, but these improvements alone will not remove all barriers to scientific progress.

Liza Litvina is a fourth year graduate student at Harvard Medical School Program in Neuroscience

Anna C. Maurer (Graphics) is a second year graduate student in the Biological and
Biomedical Sciences program at Harvard Medical School.

References

[1] Horrobin D. “The Philosophical Basis of Peer Review and the Suppression of Innovation”. The journal of the American Medical Association, 1990.

[2] Smith, R. Peer review: a flawed process at the heart of science and Journals. Journal of the Royal Society of Medecine 2006.

[3] Nature journals offer double-blind review. 18 February 2015. Nature editorial. http://www.nature.com/news/nature-journals-offer-double-blind-review-1.16931

[4] Shapin S., Shaffer S., Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (1985)

[5] Spier R., “The history of the peer-review process,” TRENDS in Biotechnology, 2002.

[6] Burnham, J. “The Evolution of Editorial Peer Review ” The journal of the American Medical Association, 1990.

[7] Sense about Science Peer Review Survey 2009. http://www.senseaboutscience.org/pages/peer-review-survey-2009.html

[8] Hojat M., Gonella J., Caelleigh A. “Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process”. Advances in Health Sciences Education, 2003

[9] McCook A, “David Vaux: Nature’s decision to add double-blind peer review is good, but could be better,” RetractionWatch.com, 2015. http://retractionwatch.com/2015/02/25/david-vaux-natures-decision-to-add-double-blind-peer-review-is-good-but-could-be-better/

[10] Grant, Bob. “Mass Retraction,” The Scientist March 27, 2015. http://www.the-scientist.com/?articles.view/articleNo/42553/title/Mass-Retraction/