Explainability and Its Limits
Explanations can help people inspect model behaviour, but they do not establish that an output is correct, causal or clinically useful.
#Explainability is one part of transparency
The FDA, Health Canada and MHRA distinguish transparency from explainability. Transparency includes clear communication about intended use, development, performance and, when available, the basis of an output. Explainability concerns making that basis understandable.
An explanation is therefore not the whole information package. Users also need the intended role, relevant evidence and known limitations. The information should fit its audience and place in the workflow, not merely make a model seem approachable.
Evidence: FDA, Health Canada and MHRA: device transparency principles
#Know what kind of explanation is shown
Ghassemi and colleagues distinguish models with directly inspectable relationships from explanations added after a complex model produces an output. Their 2021 viewpoint uses image heat maps as an example of the latter. A map highlighting regions associated with a prediction is different from a complete account of why a clinical decision is justified.
Read whether the explanation concerns one output or a pattern across many outputs, and what the method actually shows. The authors caution that an intuitive visual story can mislead. This is a dated viewpoint about the methods it examined, not a test of every current explanation tool.
Evidence: Ghassemi and colleagues: viewpoint on explainability limits (2021)
#A convincing explanation is not proof
The viewpoint separates describing model behavior from deciding whether that behavior was reasonable. It argues that explanations for individual outputs do not by themselves establish correctness or justify accepting a recommendation. An understandable relationship can also be affected by unrecognized confounding.
Treat an explanation as something to examine, not as independent evidence of clinical benefit. The transparency principles call for information about performance, risks, biases and gaps in the data as well as logic when it is available and understandable. Those questions remain even when an output has a persuasive explanation.
Evidence: Ghassemi and colleagues: viewpoint on explainability limits (2021) / FDA, Health Canada and MHRA: device transparency principles
#Use explanations alongside evaluation
Ghassemi and colleagues describe useful roles for explanation methods in troubleshooting and system audit. Their argument is not that explanations have no value, but that they should not replace careful validation. They distinguish investigating model behavior from reassurance about an individual patient-level decision.
IMDRF also calls for testing under clinically relevant conditions and evaluating human-AI interactions. Read whether the model and the way people use its information were assessed. These are general research-reading principles, not an instruction to accept or reject a clinical output. No Mynd explainability product or clinician-reviewed deployment recommendation is claimed.
Evidence: Ghassemi and colleagues: viewpoint on explainability limits (2021) / IMDRF: good machine learning practice principles (2025)
Source note
The sections above were checked against the linked sources. No clinical review has been performed. This is general research education, not a clinical guideline.