ACRL/EBSS Psychology Committee

Reproducibility

"The ability of a researcher to duplicate the results of a prior study using the same materials and procedures as were used by the original investigator. So in an attempt to reproduce a published statistical analysis, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis to determine whether they yield the same results. " (Bollen et al., 2015)

Replicability

“The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected.” (Bollen et al., 2015)

Reference

Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., Olds, J. L., & Dean, H. (2015). Social, behavioral, and economic sciences perspectives on robust and reliable science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences. https://nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf 

General Information

• 1,5000 Scientists Lift the Lid on Reproducibility Nature. May 2016
• A Manifesto for Reproducible Science Nature Human Behavior. January 2017
• Initial Steps toward Reproducible Research

The Crisis in Psychology

• The Replication Crisis in Psychology NobaProject. 2020
• Estimating the Reproducibility of Psychological Science Science. August 2015
• Psychology’s Replication Crisis is Running Out of Excuses The Atlantic. November 2018
• Over Half of Psychology Studies Fail Reproducibility Test Nature. August 2015
• APA Website Search About Reproducibility

Common Questionable Practices

• Failure to report all dependent measures
• Collecting more data after seeing whether results were significant
• Failing to report all conditions
• Stopping data collection after achieving the desired results
• Rounding down p-values
• Selectively reporting studies “that worked”
• Excluding data after looking at the impact of doing so
• Claiming to have predicted an unexpected findings
• Falsely claiming that results are unaffected by demographics
• Falsifying data

Reference: 

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953 

Best Practice Guidelines

• Transparency and Openness Promotion (TOP) Guidelines. Center for Open Sciences (2014). These are a set of 8 standards intended to be adopted by academic journals to improve reproducibility through transparency. The TOP guidelines are endorsed by over 2900 journals and organizations. 
• Rigor and Reproducibility: Principles and Guidelines for Reporting Preclinical Research. National Institute of Health (NIH). A set of 5 guidelines for reporting preclinical research.
• A Framework for Ongoing and Future National Science Foundation Activities to Improve Reproducibility, Replicability, and Robustness in Funded Research. National Science Foundation (NSF). 
• Recommendations to Funding Agencies for Supporting Reproducible Research. American Statistical Association (ASA).
• Enhancing Research Reproducibility. Federation of American Societies for Experimental Biology.
• Registered Replication Reports. Association for Psychological Science.
• Registered Reports. Center for Open Science.

What Librarians Can Do to Support Reproducibility

• Sayre, F., & Riegelman, A. (2018). The Reproducibility Crisis and the Role of Academic Libraries. College & Research Libraries, 79(1), 2. https://doi.org/10.5860/crl.79.1.2
• Sayre, F., & Riegelman, A. (2018). Replicable Services for Reproducible Research: A Model for Academic Libraries [preprint]. College & Research Libraries. https://crl.acrl.org/index.php/crl/article/view/16993
• Riegelman, A. R., & Sayre, F. (2019). Helping Science Succeed: The Librarian’s Role in Addressing the Reproducibility Crisis. Webinar for the Medical Library Association. https://conservancy.umn.edu/handle/11299/202524
• Librarians for Building Momentum for Reproducibility (conference): https://www.youtube.com/channel/UCVyLsPwIuxjXtDtRrxRi31Q/videos