by Steph Guerra
figures by Krissy Lyon

Reproducibility in science is not very sexy. Because our scientific culture generally rewards innovation over cautiousness, replicating a study conducted by others will not get a researcher a publication in a high-end journal, a splashy headline in a newspaper, or a large funding grant from the government. In fact, only an estimated 0.15% of all published results are direct replications of previous studies [1]. We tend to take published studies at face value, assuming that the data and conclusions presented are accurate. Then we build upon these studies, using them to generate new hypotheses, new experimental designs, and new clinical trials, in a cycle that repeats over and over again.

Unfortunately, in the past few years, many studies have reported that the majority of results within biomedical research cannot be replicated. The Reproducibility Project: Cancer Biology, an open-source replication project established in 2013, aims to examine this problem by attempting to replicate the results of fifty recently published landmark preclinical cancer papers. The project hopes to estimate the extent of the reproducibility problem and identify its contributing factors [1]. With the project’s first experimental results due out this fall, opinions are mixed as to how useful this study will ultimately be. Many scientists fear that the project’s open-source methodology will initiate a witch-hunt that could have negative impacts on the funding and reputations of individual investigators. Despite this apprehension, the majority of scientists agree that scientific reproducibility is lacking across the board.

The Reproducibility Problem

In 2011, pharmaceutical companies, Bayer and Amgen, reported reproducibility rates of 11% and 25%, respectively, following systematic testing of published biomedical studies [1]. Furthermore, a 2013 survey published by the MD Anderson Cancer Center in Houston found that over half of their researchers were unable to reproduce the results of a published scientific study [1] (Figure 1).  These groups rarely accused the researchers who had performed the original studies of willfully committing fraud. Rather, they identified irreproducibility as a systemic problem that occurs due to an intense environment that rewards publishing positive data. They also cited insufficient statistical analysis and immense competition among labs as possible contributing factors.  No matter the cause, irreproducibility continues to be a detriment to the scientific community.

It is estimated that the United States spends $28 billion dollars each year on preclinical cancer studies that produce results that cannot be repeated [2]. Beyond this possible financial toll, irreproducibility may waste the time of researchers each year. Since published studies are the basis of future studies, hypotheses, and clinical trials, one irreproducible paper is likely to have a ripple effect into other laboratories and companies–a potential waste of years of effort. In fact, according to an analysis published in 2012, a significant contributor to the high failure rate in oncology clinical trials is the poor quality of published preclinical data [4]. Reproducibility is necessary to filter out subtle biases and errors not caught by the original researchers. If a study is reproducible, scientists have more confidence that the broad conclusions are widely applicable.

Figure 1: Examining scientific reproducibility has been the focus of many studies, reports, and letters in recent years.

The Reproducibility Project: Cancer Biology

The Reproducibility Project: Cancer Biology was established in 2013 by the Center for Open Science, a group based at the University of Virginia aimed at promoting transparency in science. The group, in collaboration with contracted laboratories from the Science Exchange, aims to examine the reproducibility problem by systematically repeating the results of fifty pre-clinical cancer studies. According to the study leaders, the Project’s first replication study will be published this fall, with all fifty replication studies published by the end of 2017. The ultimate goal is to produce a large dataset that researchers may use to a) estimate the extent of the problem of irreproducibility and b) reexamine the current research and publishing practices that lead to irreproducibility in cancer biology [1].

These goals are not new, as previous reports have identified key factors influencing irreproducibility. In fact, in a 2013 open letter to scientists, leaders of the National Institutes of Health identified some systemic problems contributing to irreproducibility including: poor training of researchers in experimental design, increased emphasis on provocative statements over presentation of technical details, and lack of clear reporting of experimental instructions within scientific publications [3]. The pressures of our system lead to studies that are not reliable enough to withstand replication, often because the conclusions are not as broad or simple as claimed [3].

What makes the Reproducibility Project: Cancer Biology unique? Unlike previous groups, this team is conducting their study in an entirely open manner. All fifty papers under examination are listed by name on the project’s open wiki website for public viewing [5]. Each of the fifty papers will go through the Project workflow (described in Figure 2) to determine whether their key findings, often involving two to three experiments per study, can be replicated. Crucially, this process involves open collaboration with the lab that originally published the paper. Before any experiment even begins, the Project researchers create detailed protocols and publish them for review by the original lab and other relevant scientists. This uniquely transparent process aims to ensure the best possible chance of replication [1,5]. Despite the project’s best efforts, there is a multitude of concerns surrounding this endeavor.

Figure 2: Key steps within the Reproducibility Project: Cancer Biology workflow: from the initial identification of the targeted paper to final publication of a replication study.

Concerns Regarding the Reproducibility Project

These concerns are related to three valuable resources: money, time, and reputation. Thus far, the Reproducibility Project: Cancer has been awarded $1.3 million dollars plus a half million dollars’ worth of donated scientific materials, roughly $26,000 per study, to fund the replication work [5]. In our current funding climate, it is difficult to justify funding purely replicative studies, but if the insights from this paper are able to alleviate the reproducibility problem, perhaps we will be saving money in the long-term. The project has also reportedly expended a significant amount of time of the laboratories that published the fifty cancer papers. Most labs cited 20-30 email exchanges with the Reproducibility Project, in addition to up to two weeks of full-time work by a graduate student, to retrieve key procedural details of the paper [6].

More than being a potential waste of time and resources, a major concern is that if this study reports low reproducibility rates, individual reputations and public support for biomedical research may be damaged. There is reason to believe that these fears are justified. The Reproducibility Project: Psychology, a similar venture that tested one hundred psychology studies, was released at the end of August 2015 and within days, the headlines rolled in:  Nature’s “Over half of psychology studies fail reproducibility test” and The Guardian’s “Psychology research: hopeless case or pioneering field?,” among others. The public evaluation of large-scale replication projects could undermine individual, institutional, and overall field reputations quite easily.

Beyond these concerns, many scientists are worried that direct replication by contracted laboratories is not the way to examine the reproducibility problem. Given that the scientists performing the reproducibility studies may not have the same expertise as the scientists that conducted the original studies, most authors will not be surprised if their papers are not replicated. Jeff Settleman, the CEO of Calico Life Sciences and an author of two of the fifty papers, described this issue to Science using the analogy of a recipe: “You can’t give me and Julia Child the same recipe and expect an equally good meal.” Researchers at the Reproducibility Project: Cancer Biology disagree. They maintain that not only are their replicating laboratories highly specialized, but that if the recipe (in this case, the experimental methodology described in the paper), has enough detail, two different people should be able to produce the same meal [6].

By all accounts, the leadership of the Reproducibility Project: Cancer Biology is acting in good faith with a clear desire to replicate as many results as possible. It remains to be seen whether this levelheaded attitude will extend to how results are actually reported by the media, or if the study will yield its own set of sensational headlines to the detriment of the cancer research field.

In a country where biomedical research funding has been steadily declining over the past decade [8], it is natural to wonder whether devoting time, money, and reagents to such large-scale replication studies is worthwhile. There have already been many smaller studies and analyses to examine this problem, and it is unclear whether this approach will reveal any new insights. While there are concerns about how the results of The Reproducibility Project: Cancer Research will be published, there is at least one benefit: all of the researchers are bravely contributing to the discussion about our scientific culture. As Brian Nosek, the founder of the Center for Open Science and the lead author of the Reproducibility Project: Psychology study told The Atlantic, “I really hope that this isn’t a one-off, but a maturing area of research in its own right.” [8] Perhaps reproducibility is getting a little bit sexier.

Steph Guerra is a Ph.D. student studying cancer therapeutics and the co-director of Harvard’s Science in the News. 

References:

[1] Errington, et al. Science forum: An open investigation of the reproducibility of cancer biology research. eLife, December 2014. [http://elifesciences.org/content/3/e04333]
[2] Kaiser, Jocelyn. Study claims $28 billion a year spent on irreproducible biomedical research.  Science Magazine News. 9 June 2015. [http://news.sciencemag.org/biology/2015/06/study-claims-28-billion-year-spent-irreproducible-biomedical-research]
[3] Collins, Francis S. and Lawrence A. Tabak. Policy: NIH plans to enhance reproducibility. Nature comment. 27 January 2014. [http://www.nature.com/news/policy-nih-plans-to-enhance-reproducibility-1.14586#/b4]
[4] Begley, C. Glenn and Lee M. Ellis. Drug Development: Raise standards for preclinical cancer research. Nature 483, 531-533. 28 March 2012. [http://www.nature.com/nature/journal/v483/n7391/full/483531a.html]
[5] Reproducibility Project: Cancer Biology, Open Science Framework. [https://osf.io/e81xl/wiki/home/]
[6] Kaiser, Jocelyn. The Cancer Test. Science. 26 June 2015.
[7] Yong, Ed. How reliable are psychology studies? The Atlantic. 27 August 2015. [http://www.theatlantic.com/health/archive/2015/08/psychology-studies-reliability-reproducability-nosek/402466/]
[8] NIH Research Funding Trends. Federation of American Societies for Experimental Biology.

2 thoughts on “Reproduce or Bust?!: Bringing Reproducibility Back to Center Stage

  1. Nice overview of the reproducibility problem!
    Readers may be interested to learn that, along with efforts like The Reproducibility Project, a different sort of program aims to identify modifiable behaviors that contribute to irreproducibility, particularly in preclinical biomedical research. For example, the Global Biological Science Institute – GBSI, the group that came up with the $28 billion estimate cited the news article in reference 2 above (See The Economics of Reproducibility in Preclinical Research in PLoS Biology; http://tinyurl.com/p2w46g9) — recently looked at cell line authentication in preclinical research. It has long been known that misidentified human cell lines have made their way into numerous published studies. (See references 29, 30, 31 in the PLoS Biology paper.) To see if the early reports on cell line misidentification have led to a behavioral change among researchers, GBSI President Leonard Freedman, PhD and his collaborators conducted a cell authentication survey. Results, published in BioTechniques, showed a pronounced lack of attention to this matter among surveyed researchers at all career stages. (See http://tinyurl.com/o8na9al) Now that there is evidence that the problem still exists, education efforts are underway to help researchers routinely authenticate the cells used in their studies. Moreover, funding agencies are starting to require that their awardees authenticate cells annually in order to maintain their funding. Taking the lead on this effort among funders was the Prostate Cancer Foundation, whose Chief Scientific Officer, Howard Soule, PhD, was a co-author on the BioTechniques letter.
    Beth Schachter, PhD
    Science Communications Consultant
    Consultant to GBSI

  2. Whether you believe that science is in need of fixing – or that it’s working better than ever – it’s clear that change is coming. As a community, we need to work together, with the ultimate goal of increasing openness, transparency and reproducibility in science.

Leave a Reply

Your email address will not be published. Required fields are marked *