Training, Validation and Test Sets

#Describe the job, not only the label

Training or development data are used to build a prediction model. Tuning data help select settings or compare candidate models. Evaluation or test data estimate performance after those development choices. The same word, "validation", can mean tuning in one report and independent evaluation in another.

TRIPOD+AI therefore asks readers to distinguish development from evaluation and describes internal validation as evaluation within the development population, such as cross-validation or bootstrapping. Read how each dataset was actually used, rather than assuming its name establishes independence.

Evidence: TRIPOD+AI: reporting prediction-model studies (2024)

#Check whether the separation is real

TRIPOD+AI says evaluation data should be distinct from data used to train, tune or select the model, with no overlap in participants. IMDRF also asks developers to consider dependence related to patients, sites and data acquisition. Moving rows into different files is not enough if those rows still share a person or a local recording process.

Ask how participants and sites were assigned to datasets and which decisions used each dataset. These questions help expose information leakage or dependence that can make evaluation appear easier than the intended use. The appropriate design depends on the task; this page does not prescribe a universal split ratio.

Evidence: TRIPOD+AI: reporting prediction-model studies (2024) / IMDRF: good machine learning practice principles (2025)

#Read the uncertainty and the population

A performance estimate describes the model, data and evaluation method used to obtain it. TRIPOD+AI calls for reporting discrimination, calibration and clinical utility, with performance estimates and their uncertainty. An overall result can hide important differences among groups.

Evaluation data should represent the population in which the model is intended to be used. Internal evaluation can reveal overfitting, but it is not the same question as evaluation in another care setting or later period. Describe the population and the method alongside the number, not only an accuracy headline.

Evidence: TRIPOD+AI: reporting prediction-model studies (2024)

#A test is not a permanent certificate

Changing a model after seeing its evaluation results is further development. TRIPOD+AI asks authors to report model updating and the updated model's performance so readers can distinguish what was tested from what was changed. The statement avoids describing a model as permanently "validated".

IMDRF considers clinically relevant testing and ongoing monitoring across a device's life cycle. A well-separated evaluation does not by itself establish clinical benefit, regulatory permission or safety in every setting. This is general prediction-model literacy, not a checklist authorizing deployment of a Mynd clinical AI service.

Evidence: TRIPOD+AI: reporting prediction-model studies (2024) / IMDRF: good machine learning practice principles (2025)

Source note

The sections above were checked against the linked sources. No clinical review has been performed. This is general research education, not a clinical guideline.