Aim
External validation of machine learning (ML) predictive models is critical to establish the generalisability by evaluating model performance on a cohort that is distinct from that which the model was developed. This important step is not commonly performed – prohibiting clinical translation.
An enhanced ML model (LASSO + SIVS) of the Danish Knee Ligament Reconstruction Registry (DKRR) identified 4 variables (Age + 3 post-op KOOS items) which predicted risk of revision ACLR - DK3 Risk Monitoring Tool1.
The purpose of this study was to externally validate the DK3 with other National Ligament Registries both locally (Norway) and geographically remote (New Zealand) to evaluate the utility and transferability of this model across distinct cohorts.
Methods
The primary outcome measure of the original DKRR model was the probability of revision ACL reconstruction within 1, 2, and/or 5 years. For external validation, all Norwegian Knee Ligament Registry (NKLR) and New Zealand (NZ) ACL Registry patients with complete data for the four variables required for DKRR model were included.
External validation of the DK3 was performed using both original and refitted regression co-efficients. The ML analysis was then repeated using SIVS with the entire NKLR and NZ ACL Registry datasets.
Model performance was assessed using the same metrics as the DKRR study – i.e. concordance and calibration.
Results – NB Preliminary (*NZ ACL Registry analysis results pending)
In total 18,117 patients from NKLR were included for analysis (2005-2024).
Average follow-up time or time-to-revision was 8.4 (± 4.3) years and overall revision rate was 6.9%.
*New Zealand ACL Registry = N = 25,671 (2015-2024) – 55% F/Up @ 2 years; 34% @ 5 years. Revision rate 8.8%.
A representative cohort was evaluated with respect to age, sex and injury cause. Notably surgical trends (graft choice and fixation) and injury characteristics (i.e. meniscus/cartilage damage) varied between registries, potentially influencing results.
- Graft choice (DKRR = 80% HS; NKLR 50% BPTB; NZACLR 45% BPTB)
SIVS analysis identified age and Quality of Life (QoL) subscale items as the strongest predictors of ACLR revision risk - The individual QoL item(s) varied between cohorts suggesting regional differences.
Model performance demonstrated similar concordance when applied to the NKLR cohort (C = 0.73 Vs 0.75). Calibration was also similar at various timepoints.
Conclusion
Enhanced ML modelling (LASSO + SIVS) demonstrated similar performance when applied to cohorts from other National Ligament Registries, both locally (Norway) and geographically remote (NZ).
This confirms external validation of the modelling and builds on existing ML validation studies using geographically similar cohorts2.
The DK3 Risk Monitoring Tool can be used to assess ACLR revision risk at a patient-specific level and more importantly can be applied across geographically distinct cohorts using region-specific variations. Future directions suggest ML comparisons of National Ligament Registries combining datasets globally.
