When Studies Disagree

#Check whether the comparisons match

Before calling studies contradictory, compare the participants, interventions, comparators, outcomes and follow-up. Different questions can produce different estimates. Cochrane distinguishes clinical diversity from differences in methods and measurement, which can also change results.

Check the actual outcome and measurement, not only the treatment name. A short-term symptom measure and a later health event are different outcomes. A useful comparison starts with a clear account of what each study estimated, rather than choosing the conclusion that sounds most convincing.

Evidence: Cochrane Handbook, Chapter 10: meta-analysis and heterogeneity

#Compare estimates, not significance labels

A statistically significant result and a non-significant result need not represent different effects. Cochrane describes how two studies with identical effect estimates can receive different significance labels because they contain different amounts of information.

Compare the size, direction and uncertainty of the estimates on compatible measures. When assessing differences between subgroups, comparing their separate P values is not a valid test of the difference. Cochrane recommends a formal comparison and cautions that small numbers of studies can limit what such analyses establish.

Evidence: Cochrane Handbook, Chapter 12: synthesis without meta-analysis / Cochrane Handbook, Chapter 10: meta-analysis and heterogeneity

#Investigate variation without inventing an explanation

Variation may reflect different populations or interventions, differences in bias or outcome assessment, or chance. The presence of heterogeneity does not by itself identify which explanation is correct. Data extraction or entry errors can also create apparent differences.

Subgroup analyses and meta-regression can explore variation, but comparisons between studies are observational and can be confounded by other study characteristics. Explanations developed after seeing the results are especially uncertain. Read whether the analysis was planned and what outside evidence supports the proposed explanation.

Evidence: Cochrane Handbook, Chapter 10: meta-analysis and heterogeneity

#Do not settle disagreement by counting votes

Counting statistically significant studies as benefits and non-significant studies as no benefit can lead to the wrong conclusion. This approach loses information about effect magnitude, uncertainty and study size. A systematic review should explain its synthesis method even when no meta-analysis is possible.

Sometimes a meaningful pooled average is not available, or uncertainty remains after synthesis. Look for the risk of bias, missing evidence and outcome-specific certainty, rather than demanding one confident answer. This page supports reading and discussion; it does not decide which treatment an individual should choose.

Evidence: Cochrane Handbook, Chapter 12: synthesis without meta-analysis / Cochrane Handbook, Chapter 10: meta-analysis and heterogeneity / Cochrane Handbook, Chapter 13: missing evidence / Cochrane Handbook, Chapter 14: certainty of evidence

Source note

The sections above were checked against the linked sources. No clinical review has been performed. This is general research education, not a clinical guideline.