Validation is an essential part of assessing prediction models. For most studies that have access to only one data set, the emphasis is on internal validation. Internal validation is an assessment of model performance based on the data set used for development. Prediction models tend to perform best in the data set in which they were developed, and, therefore, one of the purposes of internal validation is to try and assess to what degree the estimates reflect true relationships between variables rather than the idiosyncrasies of the development data set. Teasing apart true and spurious relationships can be a problem when there are too many predictors and/or predictors are selected using a data-driven method such as stepwise regression.
The aim of our methodology was to avoid this problem by choosing a priori few predictors based on the criteria described above; see Chapter 3, Choice of predictors. These criteria do not use any information based on p-values or CIs and so avoid the problems of data-driven methods.
Internal validation methods generally consider the concepts of model discrimination and model calibration. In this context, discrimination is a reflection of how well the model differentiates between patients who do and do not ulcerate. Calibration is a reflection of how well the model assigns the relative proportions of risk categories; a poorly calibrated model would place many patients in the wrong risk category. A statistic that encompasses both these concepts is the two-component Brier score, which takes values between 0 and 1, with a perfectly calibrated model having a score of 0 and model with no calibration having a score of 1.
Given that we had several data sets, we also addressed external validation. External validation is the assessment of model performance in data sets other than the development data set and was arguably more important than internal validation because it related directly to the generalisability of our results. Performing external validation required two decisions to be made: how should it be done and which data should be used? There are six different methods of external validation for logistic regression models described by Steyerberg et al.64 We chose one method for ease of implementation and interpretation, which was simply to re-estimate the ORs of our final model in a new data set not previously used in any analysis. We then compared the ORs from the meta-analyses with those from the validation data set. The validation data set had to fulfil one mandatory criterion, that is, it had to have all the variables used in the primary model, and more than one data set fulfilled this criterion. However, one data set in particular seemed to be the natural choice as the validation data set. The reasons for this were:
The validation data set was only available for analysis at a late stage. By using this data set for validation rather than model development, work on the meta-analyses could proceed in a timely manner.
The validation data set was not accessible to those performing the meta-analyses. Analyses could be requested and aggregate results supplied from the validation data set. This ensured that the persons conducting the meta-analyses were not influenced by any validation results, which were requested only after the meta-analyses had been completed.
The persons conducting the validation analyses were not informed of the results of any meta-analyses until after the validation analyses had been completed and supplied. This ensured that they were not influenced by any meta-analysis results.
Therefore, our methods allowed the validation process to be independent of the model development process and vice versa. It is also worth noting that the use of this particular data set, which was analysed by statisticians not otherwise involved in the project, meant that the method of external validation had to be simple, as the time available to conduct the validation by these statisticians was necessarily limited. To ensure that the time required was minimal, Statistical Analysis System [(SAS); SAS Institute Inc., Cary, NC, USA; see www.sas.com] programs (version 9.3) were supplied to produce all the required analyses.
