Quality assessment of the three included articles was undertaken using an early version of the PROBAST for assessing risk of bias and applicability of prognostic model studies.15 The results of this assessment formed the structure and content of this critical appraisal covering areas including patient selection, outcomes, predictors, sample size and analysis methods of the three included studies (see Assessment of study quality).
Key similarities and differences between the models are then summarised, focussing on the implications for the robustness of the respective findings. The findings from the PROBAST assessment, in terms of areas of potential risk of bias in the included studies, are summarised in , and presented validation statistics for the studies are summarised in .
HER DOO 29
Rodger et al.9 aimed to develop and internally validated a clinical decision rule to identify a subgroup of patients at low risk of recurrent VTE, in whom OAC therapy could be stopped after 6 months.
Patient selection
Rodger et al.9 used a prospective cohort study with consecutive unselected patients from 12 tertiary care centres in four countries. The prospective cohort design ensures that predictor information can be collected blinded of patient outcome. Inclusion criteria were patients treated with low-molecular-weight heparin for at least 5 days, OAC therapy for between 5 and 7 months with an INR between 2 and 3, and with no recurrence during therapy. Patients were excluded if they would not provide consent, were aged < 17 years, had already ceased therapy, required therapy for another reason, were inaccessible for follow-up, were being treated for unprovoked recurrence or known thrombophilia. Thrombophilia testing was not performed as part of the study, but any patients with known thrombophilia prior to the start point were excluded.
The definition of unprovoked VTE used by Rodger et al.9 was based on an absence of the following provoking factors:
leg fracture or plaster cast
immobility for > 3 days
surgery using general anaesthetic (in the 3 months prior to the index event)
diagnosis of cancer (in the past 5 years).
This definition of unprovoked VTE therefore includes women who have currently, or previously, used OCs or HRT. Hormone therapy should be considered as a provoking factor for recurrent VTE; though some argue that because the effect of hormone therapy is weak it may be included within the definition.41,44 Strictly, the definition of unprovoked VTE refers to a VTE in the absence of any transient risk factors, hence why hereditary thrombophilia can be considered unprovoked VTE.45 As hormone therapy is a transient risk factor patients with a history of hormone therapy could be considered provoked. As a result there may be patients included in the model development who are at potentially lower risk than other unprovoked patients. This could lead to biased estimation of effect sizes within analyses, particularly for the effect of sex, which could lead to poor validation in external populations which may not include patients with hormone-related index events.
Outcomes
The primary outcome of the study was recurrent VTE, with associated deaths also recorded. Suspected DVT was confirmed by compression ultrasound, whereas suspected PE was confirmed by high-probability ventilation/perfusion scanning and/or spiral computed tomography. All events were adjudicated by independent physicians who were blinded to predictor information. As such, outcomes were pre-specified with the same definition and assessment used for all patients reducing the risk of detection bias, where there are differences in the determination of outcomes. The outcome definition excluded any candidate predictors again reducing the risk of detection bias and overall providing a low risk of bias with regard to outcome assessment.
Predictors
Rodger et al.9 report that information was collected on 69 potential predictors based on evidence from a pre-specified systematic review. Summary information was provided for 21 candidate predictors (with categories equating to 39 candidate predictors) including:
sex
ethnicity
age (years)
weight (kg)
height (cm)
BMI (kg/m2)
abnormal baseline imaging
abnormal baseline compression ultrasound
D-dimer (µg/l)
homocystine (mmol/l)
haemoglobin (g/l)
heterozygous for prothrombin gene mutation
factor VIII (U/ml)
factor V Leiden
ventilation/perfusion scan or pulmonary vascular obstruction result < 95%
post-thrombotic signs
history of chronic obstructive pulmonary disease or emphysema
family history of VTE
previous secondary VTE
use of OC in year before event
HRT in year before the event.
All predictors were measured before outcomes occurred, with laboratory predictors measured from samples taken while still on OAC therapy, and predictor definitions were consistent for all patients. Therefore the risk of selection bias due to differences in the characteristics of patients, and also the risk of reporting bias through differences in the reporting of predictors based on outcomes, could be considered low.
The use of a systematic review to identify potential predictors, the consistent updating of the systematic review and the assessment of all identified predictors provides confidence that there is a low risk of selection bias where patients are selected based on their characteristics, and attrition bias based on exclusions from analyses.
Rodger et al.9 categorised all continuous predictors that had a p-value < 0.2 at univariable analysis, by dichotomising at various thresholds and identifying ‘optimal’ thresholds as those with the highest chi-squared value. The process of categorisation was therefore completely data driven which often leads to reporting bias, where the most significant results are presented.46 Rodger et al.9 indicate that dichotomised predictors were used to enhance the applicability and acceptance of the proposed clinical decision rule. Although it is a valid reasoning, with researchers and clinicians looking for parsimony in any decision rule,12 the dichotomisation of continuous predictors leads to a loss of information. Dichotomisation splits patient’s risks into two groups treating patients on either side of a threshold as distinctly different, when in reality they may be very similar on the original scale.47 Best practice recommends that continuous predictors remain continuous in any analysis and to investigate non-linear associations.47–50
Sample size and patient flow
A total of 646 patients had at least one follow-up visit and were included within the analysis. Only 600 of these patients completed follow-up, with 14 lost to follow-up, 10 patients dying after their first follow-up, nine patients withdrawn and 13 patients restarting OAC therapy for another reason. There were 91 events recorded out of the 646 patients included in the analysis, which could be considered insufficient power to investigate the 36 predictors described within the article. As a rough guide at least 10 events are required for each candidate predictor being investigated to give enough power for the analysis to yield valid conclusions.51 Therefore, to investigate 36 predictors would require roughly 360 events. It is unclear from the article whether or not model building included the 36 predictors summarised within the article, or the 96 predictors for which information was recorded: roughly 960 events would be required to adequately power such an analysis.51 As the study was substantially underpowered (with a maximum of 2.5 events per predictor), the results of model building and conclusions drawn must be carefully interpreted. Predictor effects could be substantially biased, with overestimation or underestimation of both the effect size and its associated uncertainty.51
Rodger et al.9 conducted a complete-case analysis, excluding patients with any missing predictor information. There was missing predictor information quantified for five predictors included in the predictor selection procedure:
abnormal baseline compression ultrasound (events = 14/non-events = 180)
factor V Leiden (events = 0/non-events = 1)
ventilation/perfusion scan or pulmonary vascular obstruction result < 95% (events = 53/non-events = 283)
post-thrombotic signs (hyperpigmentation, edema or redness in either leg) = (events = 18/non-events = 83)
family history of VTE (events = 0/non-events = 1).
The use of a complete-case analysis for the final model may introduce attrition bias by excluding patients who have outcome data but are missing information from one or more candidate predictors. The drop in the number of events is also a cause for concern; a potential total of 85 events were missing across the five predictors, enhancing the possibility of spurious relationships being seen or important prognostic effects being missed during the predictor selection procedure. In terms of the final model, only post-thrombotic signs were included and so the complete-case analysis sample size was reduced by 101 patients, of whom 18 were events. Overall the sample size and handling of missing data within the study indicate a potentially high risk of bias within the model development.
Analysis
Rodger et al.9 used a conditional logistic regression model and selected predictors using a stepwise forward selection process. As the outcome of interest was time to recurrent VTE, a time-to-event analysis may have been more appropriate here as logistic regression does not account for the censoring of patients over time and variable lengths of follow-up. The analysis also did not consider the potential heterogeneity across the three tertiary centres from four different countries, stratification by centre or country would take into account any heterogeneity in the baseline risk of recurrence within these potentially different populations.52
Predictors were only included in multivariable analysis if univariable analysis yielded a p-value < 0.2, though univariable results were not presented in the article. This exclusion of candidate predictors from multivariable analysis was therefore completely data driven based on univariable results, which could lead to potential bias in results because predictors which may be important in combination (e.g. in a multivariable model) were not considered for multivariable analysis. Univariable analyses are not recommended for decisions about inclusion criteria in a multivariable model.53
The use of a forward selection procedure in model development can lead to overoptimism in regression coefficients (betas) and therefore a method such as shrinkage or bootstrapping should be used to account for this optimism. Rodger et al.9 did not use methods to account for optimism in their analyses and as such there is a risk that the performance of their proposed model could be weaker when applied to a new population.
Rodger et al.9 decided to split patients into two groups based on sex in a post-hoc analysis because a specified low-risk group (< 3% annual risk of recurrence) could not be identified. Post-hoc subgroup analyses such as this are often not considered in any assessment of study power. Stratifying by sex reduced the number of events for females and males to 28 and 63 events, respectively, creating similar issues to those discussed above in the estimation of regression coefficients within the model.51
Five clinical decision rules were developed for women and two for men, the final decision rule was selected based on criteria including the classification of performance; the proportion of patients identified as low risk; the face validity of the model; ease of use of the model; and more parsimonious models. The performance of decision rules for men was considered poor, particularly in identifying a low-risk group of men which could be considered for cessation of therapy, as such the study recommended that men continue with OAC therapy. The final decision rule for women included predictors for:
post-thrombotic signs:
hyperpigmentation
oedema
redness in either leg
D-dimer level ≥ 250 µg/l
BMI ≥ 30 kg/m2
aged ≥ 65 years.
Rodger et al.9 suggested that any female patient with fewer than two of these predictors could potentially safely cease OAC therapy after 5–7 months of initial OAC therapy for an unprovoked VTE. The specification of the model is not described in full, with regression coefficients presented for the final model only, but without any measurement of variability such as a standard error or 95% CI for the coefficients. It is unclear if or how the decision rule is related to the regression coefficients, though it may be that a rounding of the coefficients to the nearest integer was used. There is no indication of the level of risk associated with a particular score using the proposed decision rule; for example, what the risk of recurrence at 1 year post cessation of therapy is for a female with three of the included factors.
The decision rules developed were internally validated using split-sample cross-validation. Five hundred subsamples, half the size of the study sample, consisting of randomly selected patients from the population were used to assess rule performance. The mean annual recurrence risk predicted by the clinical decision rule was recorded within each subsample and showed that for all subsamples the decision rule identified a low-risk group of women with mean annual risk of recurrence between 0% and 3%, suggesting that the rule performed well in internal validation. No measure of performance in terms of calibration or discrimination was presented, and no external validation of the rule was conducted. An external validation of the clinical decision rule is currently being undertaken comparing use of the decision rule to decide on cessation of therapy versus standard therapy.27,28
Summary
Overall there are significant concerns over the robustness of the proposed clinical decision rule. The testing of many more predictors than the study was powered for could lead to potentially spurious predictor effects. The data-driven dichotomisation of continuous predictors does not allow for non-linear effects, and instead suggests a constant effect in patients at either side of the chosen threshold.48 The effects of missing predictor information were not assessed which could have resulted in different conclusions had there been more data available. Potentially inappropriate analyses were performed, without accounting for heterogeneity in baseline risk across differing populations and performing post-hoc subgroup analysis. The decision rule was poorly presented, with a lack of explanation linking regression coefficients with the decision rule. Furthermore, there was no reporting of uncertainty surrounding coefficients and no description of calibration or discrimination given. Therefore, there is substantial concern that the proposed decision rule would not perform as presented when applied in a new population and examined in new data independent to that used to develop the model.
Vienna prediction model2,42
Eichinger et al.2,42 aimed to develop and internally validate a prognostic model to improve VTE recurrence risk prediction for patients following a first unprovoked VTE.
Patient selection
Eichinger et al.2,42 also used a prospective cohort design with consecutive unselected patients recruited from four thrombosis centres in Vienna, between July 1992 and August 2008. Similar to Rodger et al.,9 a prospective cohort design ensures that predictor information can be collected blinded of patient outcome. Patients were included if they were at least 18 years old and had been treated with OAC therapy for at least 3 months for a first unprovoked VTE.
The definition of unprovoked VTE used by Eichinger et al.2,42 was based on an absence of the following provoking factors:
surgery
trauma
pregnancy
hormone intake
deficiency in antithrombin, protein C, protein S
presence of lupus anticoagulant, or
cancer.
This definition of unprovoked VTE therefore follows a standard definition based on an absence of any transient risk factors. As a result the study population should be representative of the unprovoked VTE population and therefore provides a low risk of bias in the model development.
Outcomes
The main outcome of the study was recurrent VTE, with suspected DVT confirmed by venography or colour duplex ultrasonography. Suspected PE was confirmed by ventilation/perfusion scan and/or spiral computed tomography. All events were adjudicated by a committee of independent radiologists. Detection bias was limited by pre-specification of outcome definitions, with the same definition and assessment used for all patients, meaning systematic differences in the determination of outcomes were avoided. Outcomes were pre-specified with the same definition and assessment used for all patients reducing the risk of differences in the diagnosis and reporting of outcomes. The outcome definition excluded any candidate predictors and was also determined blind to any predictor information, again, reducing the risk of detection bias and overall resulting in a low risk of bias based on the study outcomes.
Predictors
Eichinger et al.2,42 pre-specified a selection of clinical and laboratory predictors based on criteria including independent confirmation of the association with risk of recurrence (literature), simplicity of assessment and reproducibility. Using pre-specified candidate predictors reduces the risk of bias through unnecessary investigations, limiting the number of hypothesis tests and therefore reducing the chance of multiple testing issues. Candidate predictors included:
All predictors were measured blinded to outcome data (with laboratory predictors measured at cessation of OAC therapy), and predictor definitions were consistent for all patients. These methods reduce the risk of selection bias due to differences in the patient characteristics and also the risk of bias in the reporting of predictors based on outcomes.
Eichinger et al.2,42 investigated linear forms of all continuous predictors including age, BMI and D-dimer level. The study authors also investigated a dichotomisation of BMI based on the standard threshold for obesity (BMI > 30 kg/m2) used in clinical practice. The study therefore avoided introducing bias from data-driven methods of categorising continuous predictors, but this approach can still lead to a loss of information by splitting patients risk into distinct groups.47
Sample size and flow
A total of 929 patients were included within the analysis with 176 recurrent events being recorded. Given that Eichinger et al.2,42 investigated a total of eight predictors (15 predictors including categorisations of predictors), the number of events seen could be considered sufficient given the rule of thumb of 10 events per predictor.51 As the study could be considered suitably powered (with a maximum of 12 events per predictor), the inclusion of predictors and their effects are likely to be at low risk of bias, with effect sizes and associated uncertainty likely to be reliably estimated.51
Eichinger et al.2,42 conducted a complete-case analysis, excluding patients with any missing predictor information. There was missing predictor information quantified for five predictors included in predictor selection, but no indication of the number of associated events:
Complete-case analysis introduces attrition bias through excluding patients with outcome data because they have missing information related to one or more candidate predictors. The drop in the number of events could have an impact on the regression coefficients and selection of predictors for inclusion within the final model, particularly for peak thrombin with a third of patients missing predictor information. Any analyses including peak thrombin would exclude a third of the study population, markedly reducing sample size and causing issues with the estimation of predictor effects. As D-dimer is the only predictor included in the final model, the complete-case analysis would have reduced the sample size by 97, though it is unclear how many events this represents and therefore could present a high risk of bias in the results presented. Overall the sample size and handling of missing data within the study indicate a potentially high risk of bias within the model development.
Analysis
Eichinger et al.2,42 used a Cox proportional hazards model and selected predictors using a forwards stepwise selection process. A Cox regression model accounts for the censoring of patients over time and variable length of follow-up in a time-to-event analysis, making it an appropriate choice for the recurrent VTE outcome. No stratification by centre was performed as part of the analysis, which did not account for potential heterogeneity in the baseline risk of recurrence for patients from the four different thrombosis centres. Failing to account for differences in baseline risk (or at least to investigate whether or not stratification is necessary), could lead to biased predictor effects which may not be replicated when the model is applied in a new population.52
Eichinger et al.2,42 first fitted a saturated model using all clinical predictors and then performed forward selection to investigate laboratory predictors (using an inclusion threshold of p-value < 0.5). To account for optimism associated with the stepwise selection procedure, Eichinger et al.2,42 evaluated the statistical significance of included predictors using bootstrap zero-corrected 95% CIs. From 1000 bootstrap resamples of the population, only factors for which the 95% CI did not overlap zero were included in the model. Further to this, Eichinger et al.2,42 applied a shrinkage factor (calculated by bootstrap resampling), to the final beta coefficients to adjust for optimism which may affect the models performance in new study populations. The extensive use of methods to account for optimism in their analyses suggests that performance of the model is unlikely to be affected when applied to a new population.
Two prognostic models were presented within the study including the following predictors:
sex (female as the reference category)
site of index event (distal DVT as the reference category)
peak thrombin (included in the first model)
D-dimer (included in the second model).
Both models included sex and site of index event as predictors, but during initial predictor selection peak thrombin was identified as significantly prognostic and D-dimer did not enter the model [hazard ratio (HR) 1.21, 95% CI 0.87 to 1.53; p = 0.622]. The authors then decided to evaluate a model including D-dimer without peak thrombin, and in this model D-dimer was a significant predictor of recurrent VTE. Eichinger et al.2,42 chose to take forward the model including D-dimer levels as a final model because ‘D-dimer is a well-standardised and widely established parameter’. The ad hoc selection of predictors may be a cause for concern as it was not pre-specified that D-dimer was to be included and there was strong evidence against the prognostic value of D-dimer compared with peak thrombin levels. However, as discussed by the authors, the inclusion of D-dimer within the model could potentially improve the implementation of the model, as D-dimer is an established predictor more readily measured in practice. Prognostic models should aim to include predictors with standard definitions, which are easily available at the time the model is intended for use; however, the performance of the model may have been significantly improved by the inclusion of peak thrombin as a predictor (which showed strong evidence of prognostic value). However, it is also worth noting the substantial number of missing data for peak thrombin (300/929 patients missing peak thrombin data), which could have influenced the estimated effect of peak thrombin within the complete-case analysis performed.
Eichinger et al.2,42 proposed a nomogram based on the final model including sex, site of index event and D-dimer as predictors. The nomogram can be used to calculate patient’s cumulative recurrence rate at 12 and 60 months from cessation of therapy, with estimated 95% CIs. The relationship between the regression coefficients of the model and the simplified nomogram is not explicitly stated, though it is suggested that the coefficients are first multiplied by the estimated shrinkage factor. No estimate of baseline risk is provided and therefore it is only possible to estimate patients predicted risk of recurrence at the specified time points within the study (1 and 5 years). The use of estimated recurrence risk at specific time points (with associated uncertainty), improves on the HER DOO 2 model9 by allowing practitioners to apply their own judgement based on the predicted risk, using current guidelines and patient consultation, to make an informed decision on treatment strategy for the individual patient.
Internal validation of the model was performed using bootstrap cross-validated risk scores. Patients were randomly drawn with replacement from the original sample, to make a new bootstrap sample of 929 patients. Eichinger et al.2,42 re-evaluated their model within this new bootstrap sample and validated their model in the sample of patients that were not selected from the original study data. The process was repeated 1000 times and an average risk score for each patient was calculated from the 1000 replications. Using these averaged risk scores the performance of the model (within the validation subsets) was assessed using the AUC, which is a measure of model discrimination. For recurrence risk at 12 and 60 months from cessation of therapy the optimism adjusted (bootstrapped) AUC was estimated at 0.674 and 0.646, respectively, indicating moderate discrimination, which suggests moderate ability of the model to separate groups of patients such as high- and low-risk patients (where AUC = 1 represents perfect discrimination). The study also reported an apparent c-statistic (assessing discrimination across all time points, and without adjustment for optimism) of 0.651 for the developed model as a measure of discrimination (where c-statistic = 1 represents perfect discrimination), which suggests a small reduction in model performance after accounting for optimism. The Vienna model also calculated a bootstrap optimism-adjusted calibration slope (or uniform shrinkage factor), which showed moderate calibration performance of 0.88 (with 1 indicating perfect calibration). This shrinkage factor was also used to adjust the predictor effect values in their final model, to adjust for the overoptimism. However, the performance of a model measured in the same data set used to develop the model will always be biased, showing greater performance than can be expected in an external setting.
An external validation of the Vienna prediction model is currently being undertaken, which should provide a more reliable indication of model performance in a new patient population. It should also be noted that the internal validation of the Vienna prediction model relates to the multivariable Cox regression model developed, and it is unclear whether or not internal validation of the simplified nomogram was conducted.
Summary
In summary, there was a moderate risk of bias associated with the Vienna predication model, mainly because no external validation has yet been performed. Model development itself was undertaken well. Patient selection avoided inappropriate exclusions and outcomes were defined consistently for all patients and blinded to predictor information. Continuous predictors were assessed in their linear form as opposed to categorising, therefore avoiding a loss of information. Overoptimism in the estimated regression coefficients was accounted for by using bootstrapping methods to adjust these coefficients.
However, there were also some areas for concern which could have introduced bias into the model development. The analysis did not investigate potential heterogeneity in the baseline risk of recurrence in the different populations from the four thrombosis centres used, potentially affecting the models applicability to a new population. The study authors made an ad hoc decision to include D-dimer as a predictor in the model, despite its lack of prognostic value during predictor selection. There was strong evidence of an important effect for peak thrombin, which meant D-dimer was originally excluded. The decision to include D-dimer was based on practical implications for model use, as it is a more established predictor. There were also some issues with missing predictor information, with marked reductions in data for both D-dimer and peak thrombin, and no information on the number of events excluded from complete case analyses investigating these predictors. Furthermore, it was not clear how the nomogram was created and whether or not the internal validation of the nomogram was examined.
Overall the Vienna prediction rule was presented well and classed as low risk of bias in terms of model development. External validation is, however, now essential for the proposed nomogram. If performance was found to be acceptable, the model would allow individual prediction of recurrence risk limited to the two specified time points. The model was presented in a nomogram which may facilitate uptake of the model in practice, though this format limits the precision available (e.g. in specifying a patient’s exact D-dimer level). The predicted recurrence rate is provided with associated uncertainty (95% CI), which allows both clinician and patient to make an informed decision regarding treatment.
Finally, the extension to the Vienna model aimed to allow prediction of recurrence risk at further time points post cessation of therapy by measuring D-dimer over time.42 Measurements were made at 3, 9 and 15 months post cessation of therapy, and three more nomograms were developed to allow risk prediction using the Vienna model at these time points. D-dimer levels did not vary over the observation time and the associated HRs remained very similar (only the point estimate slightly decreased over time).42 A web-based calculator allows users to predict recurrence risk at any time between baseline (3 weeks) and 15 months post cessation of therapy.42 The model was adjusted for optimism using leave-one-out resampling to calculate shrinkage factors for 3, 9 and 15 months of 0.79, 0.81 and 0.7, indicating moderate calibration of the model at all time points but reduced performance compared with the original Vienna model (optimism-adjusted calibration slope = 0.88). In terms of discrimination performance (for 5-year predictions) at each time point, optimism adjusted AUC values were 0.61, 0.61 and 0.58, representing a small reduction in performance compared with the original model (AUC = 0.646).2,42 However, although the earlier Vienna model has recently been externally validated, this model has not been externally validated to date.
DASH score41
Tosetto et al.41 aimed to develop and internally validate a clinical prediction guide to stratify unprovoked VTE patients by their risk of recurrence and identify those suitable for long-term OAC therapy. The study performed a meta-analysis of IPD from seven prospective studies (see Main study and patient characteristics), so as to alleviate issues of statistical power often encountered in single prospective study.
Patient selection
Tosetto et al.41 used IPD from seven prospective cohort studies with consecutive unselected patients described by Douketis et al.11 previously. The prospective cohort design as used in the previous studies ensures that predictor information can be collected blinded of patient outcome. Patients were included if they were at least 18 years old and had been treated with OAC therapy for at least 3 months for a first unprovoked VTE. Patients were excluded if follow-up ended before D-dimer measurement, or if they had a distal DVT; only proximal DVT and PE were included as valid index sites.
The definition of unprovoked VTE used by Tosetto et al.41 was based on an absence of the following provoking factors:
This definition of unprovoked VTE therefore includes women who have currently, or previously, used OCs or HRT. As described for the Rodger et al.9 study, hormone therapy is sometimes included in the definition of unprovoked VTE because the effect of hormone therapy is weak;41,44 however, hormone therapy should be considered as a provoking factor. The definition of unprovoked VTE refers to a VTE in the absence of any transient risk factors, hence why hereditary thrombophilia can be considered unprovoked VTE.45 As hormone therapy is a transient risk factor patients with a history of hormone therapy could be considered provoked. This could lead to effect sizes within the analyses being incorrectly estimated, particularly the effect of sex, because there may be subgroups of patients included in the model development who are at potentially lower risk than other unprovoked patients.
Outcomes
The primary outcome of the study was recurrent VTE, with associated deaths also recorded. All suspected outcomes were objectively confirmed and independently adjudicated.11 Outcomes were pre-specified with the same definition and assessment used for all patients, therefore reducing the risk of differences in the determination of outcomes. The outcome definition excluded any candidate predictors and outcomes were also determined blind to any predictor information again reducing the risk of detection bias.
Predictors
Tosetto et al.41 used a backward elimination approach, starting with a saturated model including the following candidate predictors:
sex
age (years)
site of index event (DVT alone, or DVT and PE)
D-dimer (ng/ml)
hormone use at time of VTE (women)
previous history of cancer.
All predictors were measured blinded to outcome data (with D-dimer measured 3–5 weeks after cessation of OAC therapy), and predictor definitions were consistent for all patients. These methods reduce the risk of selection bias due to differences in the patient characteristics and also the risk of bias in the reporting of predictors based on outcomes.
Tosetto et al.41 also stratified their analyses by source study to allow for potential heterogeneity in the baseline risk of recurrence within these seven different populations. The adjustment for underlying differences in the source study populations may make the final model more robust when applied within a new population, not used in the development process.
Tosetto et al.41 categorised all continuous predictors, creating a dichotomisation of D-dimer (normal vs. abnormal), while categorising age into quartiles. Quartiles of age were used to control for a non-linear relationship between patient age and recurrence risk. The study may have therefore introduced bias from categorisation of continuous predictors, which can lead to a loss of information by separating patients risk into distinct groups.47 Categorisation of age appeared to be data driven, whereas dichotomisation of D-dimer was likely based on the instrument used (though it was not stated), both inducing the risk of reporting biases.
Sample size and flow
A total of 1818 patients were included within the analysis with 239 recurrent events being recorded. Given that Tosetto et al.41 investigated a total of six predictors (14 predictors including categorisations of predictors), the number of events seen could be considered sufficient given the rule of thumb of 10 events per predictor.51 As the study could be considered suitably powered (with a maximum of 17 events per predictor), the inclusion of predictors and their effect estimates may be considered to be at low risk of bias. Predictor effects are likely to be at low risk of bias, with effect sizes and associated uncertainty likely to be reliably estimated.51
There was no missing predictor information in the analyses performed by Tosetto et al.41 and therefore it was not necessary to allow for the effects of potential attrition bias. The previous two studies have both suffered with missing predictor information and conducted complete case analyses, potentially introducing attrition bias into their analyses. Tosetto et al.41 were therefore able to use all patient data in their analyses and thus statistical power remained appropriate to assess all predictors for inclusion. However, the study also used a selection procedure meaning more predictors were considered, resulting in a proportion of missing predictor data. The DASH model considered predictors including BMI, for which only 802 out of 1818 patients had complete predictor information, which may have effected the ability to detect a BMI effect. Overall the sample size and handling of missing data within the study resulted in a low risk of bias associated with the model development.
Analysis
Tosetto et al.41 used a Cox proportional hazards model and selected predictors using a backwards elimination process. A Cox regression model accounts for variable lengths of follow-up and the censoring of patients over time and in a time-to-event analysis, making it an appropriate choice for the time to recurrent VTE outcome. The analyses were stratified by source study to allow for the differences in the baseline risk across the seven different study populations, therefore avoiding biased predictor effects which may improve the external performance of the model.52
Tosetto et al.41 first fitted a saturated model using all clinical predictors, and then performed backward elimination to investigate candidate predictors (using an exclusion threshold of p-value > 0.1). To account for excessive optimism associated with the backward selection procedure, Tosetto et al.41 evaluated this using a heuristic formula and linear shrinkage by bootstrapping. A correction factor was calculated and applied to the final beta coefficients to adjust for optimism which may affect the models performance in new study populations. The use of shrinkage methods to account for overoptimism in their selection process and analyses provides a low risk that the performance of their proposed model could differ when applied to a new population.
The final DASH score was developed by multiplying the regression coefficients by the calculated correction factor, doubling these coefficients and rounding them to the nearest integer. Giving a final model included the following predictors:
abnormal D-dimer (post therapy), score = +2
age (≤ 50 years), score = +1
sex (male), score = +1
hormone use (at time of index event, in women), score = –2.
The proposed score can be used to calculate patients’ cumulative recurrence rate at 1, 2 and 5 years from cessation of therapy, with estimated 95% CIs. Despite stratification for source study in the analysis, there is no reported estimate of baseline risk and therefore patients’ predicted risk of recurrence can only be estimated at the specified time points presented. The use of estimated recurrence risk at specific time points (with associated uncertainty) is similar to that presented for the Vienna prediction model.2,42 This is a substantial improvement compared with the HER DOO 2 model9 as it allows physicians and patients to make an informed decision on treatment duration for the individual patient.
Internal validation of the model was performed using a bootstrap procedure similar to that described for the Eichinger et al. study.2,42 Patients were randomly drawn with replacement from the original sample, to make a new bootstrap sample of 1818 patients. Tosetto et al.41 then re-estimated the DASH score within this new bootstrap sample, to confirm the recurrence rate and associated CI for DASH score of < 1 (identified as having an annual recurrence risk < 5%). The process was repeated 500 times and an average risk of recurrence was calculated to be less than the agreed 5% annual recurrence risk. Apparent c-statistics (which represent the discriminatory performance within the development data without adjustment for optimism using, for example, bootstrapping) were between 0.71 and 0.72 for the score and model (beta terms), respectively, indicating moderate discrimination ability even for the simplified score for use in practice; however, apparent performance is likely to be optimistic. The DASH model also provided a bootstrap optimism-adjusted calibration slope (or uniform shrinkage factor) as the Vienna model did, which also showed strong calibration performance of 0.97 for the DASH model (with 1 indicating perfect calibration). The shrinkage factor was then used to adjust the predictor effect values for overoptimism in the final model. However, the performance of a model measured within the development data set is likely to be biased, indicating stronger performance than could be expected in a new population. The use of IPD meta-analysis in the derivation of the model and stratification for source studies may make the DASH score more robust to departures from the development population’s characteristics, but external validation should be sought to identify the true performance of the model in a new patient population.
Summary
In summary the DASH score proposed by Tosetto et al.41 could be considered at moderate risk of bias, mainly due to the lack of external validation and the categorisation of continuous predictors in the model development. Many aspects of model development were done well. Patient selection avoided inappropriate exclusions and outcomes were defined consistently for all patients and blinded to predictor information. Stratification was used in analyses to account for heterogeneity in the baseline risk of recurrence across the source studies. Missing predictor information was not an issue in the model development, avoiding attrition bias and preserving statistical power. Overoptimism in the selection of predictors and estimated effects was accounted for by a correction factor calculated by bootstrapping. Reporting of the final score was clear with recurrence risks associated with particular scores presented including uncertainty.
However, there were also some areas for concern which could have introduced bias into development of the DASH score. There were issues with categorisation of continuous predictors, which could lead to a loss of important prognostic information. Furthermore, and most importantly, there was no external validation.
External validation is therefore now essential. The DASH score provides individual recurrence risk prediction at specific time points post cessation of therapy. If externally validated and found to perform well, then the score could be useful in practice as the included predictors are well defined and readily available at the time the decision rule would be applied. However, the true performance of the DASH score within a new patient population is unclear given the lack of internal and external validation statistics. Any physician or patient using the DASH score should therefore interpret the predicted recurrence risk with care, and the included 95% CIs are importantly presented within the study, allowing informed decision-making regarding treatment.
Comparison of included studies quality
All studies performed suitable patient selection avoiding inappropriate exclusions, used appropriate study designs, pre-specified outcomes and assessed outcomes blinded to predictor information, giving low risk of bias across all studies. All studies recruited patients from different centres or countries; however, only one (Tosetto et al.41) stratified by source in their analyses. Stratification accounts for heterogeneity in the baseline recurrence risk in different patient groups. Ignoring the clustering of patients within centres or countries could lead to poor model calibration,52 where model predictions do not closely fit observed recurrence rates, and could diminish performance in a new setting.
The three studies investigated a wide variety of candidate predictors, including clinical and laboratory predictors. Eichinger et al.2,42 avoided the categorisation of continuous candidate predictors (see ), Tosetto et al.41 investigated patient age in quartiles, but pre-specified the analysis to allow for non-linear associations between age and recurrence risk. Rodger et al.,9 in contrast, performed chi-squared testing to identify the optimal threshold to dichotomise every continuous predictor under consideration. The data-driven nature of the analysis incites reporting biases where the optimal thresholds are reported without any clinical meaning. Dichotomisation of continuous predictors is also methodologically poor, as it seeks to separate patients risk into two categories treating those above and below the threshold as having different constant risks, which is unrealistic in practice.48
The HER DOO 2 model development by Rodger et al.9 was markedly underpowered, having collected information on 69 predictors and assessed at least 36 candidate predictors, with only 91 recurrent events. Given a rule of thumb based on at least 10 events per candidate predictor to be investigated,51 Rodger et al.9 only had 2.5 events per predictor, indicating a lack of power that could lead to biased estimates of predictor effects. Following the same rule, the other two studies had sufficient numbers of events to assess the predictors of interest with appropriate statistical power (see ).
All included studies suffered from some degree of missing predictor information (either in predictor selection or final model predictors) and used a complete-case analysis to overcome this issue within the final models (see ). No methods to assess the impact of this missing predictor information were used (i.e. an imputation procedure), and in the study by Eichinger et al.2,42 the number of missing recurrent events was not reported, so no assessment of the statistical power could be made accurately. Attrition bias can lead to unbalanced groups of patients and exclusion of patients reduces sample size making estimation of predictor effects biased and performance of the model specific to a subgroup of the population for whom information was not missing (there may be a risk of bias due to the nature of the missing data).
Two studies used bootstrapping and shrinkage methods to adjust predictor coefficients for over-optimism (see ),2,41,42 whereas the HER DOO 2 development did not account for optimism in predictor estimates.9 The use of optimism correction methods provides a lower risk of biased, unrealistic predictor effects, and should ensure the model performance is more consistent in a new patient population. Another methodological issue relating to model performance is validation; internal validation was performed across all of the studies, but only one has since been external validated (see ).2,42 Internal validation was reported in terms of both calibration and discrimination within the DASH41 and Vienna models2,42 (though not for the simplified nomogram), whereas Rodger et al.9 presented neither (see ): both calibration and discrimination are vitally important performance statistics for any prognostic model. External validation is the true indication of model performance, as a model validated within its development data set will always give optimistic performance statistics.18 The Vienna model has now externally validated (see Relevant studies identified after the search cut-off dates),2,42 but issues remain because (i) validation was shown to be lower than expected and uncertainty was high;54 (ii) a new Weibull model component was added, which itself requires additional validation; (iii) the nomogram version of Vienna, which is the most used, was not validated; and (iv) validation was not made by independent authors to the original model development. Thus, until further external validation is undertaken, the true performance in new populations cannot be ascertained. Further external validation studies are currently being undertaken to validate both the HER DOO 2 decision rule8,27,28 and the Vienna prediction model, which will provide a true indication as to the overall performance (in terms of calibration and discrimination) of these models in new patient populations where they are intended for use.
Finally, the application of the proposed models was described in various ways across the studies. Both the Vienna prediction model2,42 and the DASH score41 were presented well, with an indication of how the predictors are combined to calculate a patient’s recurrence risk at a specific time point (see ). Both provided cumulative recurrence rates at specific time points after cessation of therapy including an estimate of the uncertainty surrounding these estimates (95% CIs). This information could be used to direct the decision-making process, informing clinicians and patients of the individual’s level of risk and therefore allowing individualised treatment strategies. Conversely, the HER DOO 2 model9 derived a clinical decision rule splitting patients into those with less than two predictors (from their model) and those greater than two predictors, suggesting that one group could continue OAC therapy, while the other could safely stop. Rodger et al.9 did not report individuals risk at specific time points, only that fewer than two predictors would indicate a < 3% annual risk of recurrence. This therefore does not allow clinicians or patients to make decisions based on their preference of recurrence risk threshold, limiting the applicability of the decision rule.
