Reproducibility and Provenance

#State which kind of repeatability is meant

The National Academies' 2019 report uses reproducibility for consistent results obtained with the same input data, computational steps, methods, code and conditions. It uses replicability for consistent results across studies that each collect their own data to answer the same question. It notes that terminology differs across fields.

Read the definition used by the authors. Re-running an analysis and gathering new observations are different checks. This page follows the report's computational distinction, not a claim that every scientific discipline uses the terms in the same way.

Evidence: National Academies (2019): reproducibility and replicability / National Academies (2019): reproducibility report summary

#The path from measurement to analysis matters

The report describes data definition, collection, review and curation as stages that can affect reproducibility and replicability. Its clinical example traces data from a raw measurement through interpretation to the coded analytic file, with cleaning and transformation decisions along the way.

Read where the data came from and how they became the input to the analysis. A final spreadsheet alone may not expose the choices that shaped it. Provenance here means that documented path, rather than a guarantee that the source or analysis is correct.

Evidence: National Academies (2019): reproducibility and replicability

#Available artifacts and checked results are different

The report distinguishes making data, code and methods available from another researcher actually recomputing the results. Its summary notes that proprietary or nonpublic artifacts create challenges for transparency, and calls for reproducibility checks to respect ethical and legal limits.

Read what is available, under what conditions, and what was actually checked. Public release of sensitive participant data is not implied by a transparency goal. Mynd has not reproduced a study's analysis or verified its code merely by citing it here.

Evidence: National Academies (2019): reproducibility and replicability / National Academies (2019): reproducibility report summary

#A disagreement calls for investigation

The National Academies explains that non-replication can arise from different sources, including previously unrecognized variation as well as errors or bias. It says that identifying the reason requires further investigation and that consistency should be considered alongside the uncertainty of the findings.

Read the methods, context and uncertainty before treating a repeat study as a simple pass or fail. Matching computations do not alone prove a sound design, while a differing result does not by itself prove misconduct. This is research literacy, not a verdict on a named study.

Evidence: National Academies (2019): reproducibility report summary / National Academies (2019): reproducibility and replicability

Source note

The sections above were checked against the linked sources. No clinical review has been performed. This is general research education, not a clinical guideline.