One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

Contents

• The National Academies of SCIENCES • ENGINEERING • MEDICINE
• COMMITTEE ON REPRODUCIBILITY AND REPLICABILITY IN SCIENCE
• BOARD ON BEHAVIORAL, COGNITIVE, AND SENSORY SCIENCES
• COMMITTEE ON NATIONAL STATISTICS
• NUCLEAR AND RADIATION STUDIES BOARD
• BOARD ON MATHEMATICAL SCIENCES AND ANALYTICS
• COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
• BOARD ON RESEARCH DATA AND INFORMATION
• COMMITTEE ON SCIENCE, ENGINEERING, MEDICINE, AND PUBLIC POLICY
• Acknowledgments
• Preface
• Executive Summary
• Summary
  • THE ROLE OF REPRODUCIBILITY AND REPLICABILITY IN SCIENCE
  • DEFINITIONS
  • REPRODUCIBILITY
  • REPLICABILITY
  • CONFIDENCE IN SCIENCE
• 1. Introduction
• 2. Scientific Methods and Knowledge
  • WHAT IS SCIENCE?
  • CORE PRINCIPLES AND ASSUMPTIONS OF SCIENTIFIC INQUIRY
  • STATISTICAL INFERENCE AND HYPOTHESIS TESTING
• 3. Understanding Reproducibility and Replicability
  • THE EVOLVING PRACTICES OF SCIENCE
  • DEFINING REPRODUCIBILITY AND REPLICABILITY
  • PRECISION OF MEASUREMENT
  • VARIATIONS IN METHODS EMPLOYED IN A STUDY
  • RIGOR AND TRANSPARENCY
• 4. Reproducibility
  • WIDESPREAD USE OF COMPUTATIONAL METHODS
  • ASSESSING REPRODUCIBILITY
  • THE EXTENT OF NON-REPRODUCIBILITY
  • SOURCES OF NON-REPRODUCIBILITY
• 5. Replicability
  • ASSESSING REPLICABILITY
  • THE EXTENT OF NON-REPLICABILITY
  • SOURCES OF NON-REPLICABILITY
• 6. Improving Reproducibility and Replicability
  • STRENGTHENING RESEARCH PRACTICES: BROAD EFFORTS AND RESPONSIBILITIES
  • EFFORTS TO IMPROVE REPRODUCIBLITY
  • OVERCOMING TECHNOLOGICAL AND INFRASTRUCTURE BARRIERS TO REPRODUCIBILITY
  • EFFORTS TO IMPROVE REPLICABILITY
• 7. Confidence in Science
  • RESEARCH SYNTHESIS
  • GEOSCIENCE
  • GENETICS
  • PSYCHOLOGY
  • SOCIAL SCIENCE RESEARCH USING BIG DATA
  • PUBLIC PERCEPTIONS OF REPRODUCIBILITY AND REPLICABILITY
• References
• Appendix A. Biographical Sketches of Committee Members and Staff
• Appendix B. Agendas of Open Committee Meetings
• Appendix C. Recommendations Grouped by Stakeholder
• Appendix D. Using Bayes Analysis for Hypothesis Testing
• Appendix E. Conducting Replicable Surveys of Scientific Communities

Suggested citation:

National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.

Digital Object Identifier: https://doi.org/10.17226/25303

Library of Congress Control Number: 2019946492

Additional copies of this publication are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.

Printed in the United States of America