Machine Learning based Analysis for Radiomics Features Robustness in Real-World Deployment Scenarios

TL;DR

This study evaluates the robustness of radiomics-based machine learning models across various MRI protocols and segmentation strategies, highlighting the importance of protocol-invariant features for reliable clinical deployment.

Contribution

It systematically investigates the impact of distribution shifts on model performance and proposes robust feature selection and dataset augmentation to enhance reliability in real-world scenarios.

Findings

Protocol-invariant features maintain high F1-scores (>0.85) under distribution shifts.

Dataset augmentation improves uncertainty estimates and calibration by 35%.

Models trained on all features degrade by 40% under protocol changes.

Abstract

Radiomics-based machine learning models show promise for clinical decision support but are vulnerable to distribution shifts caused by variations in imaging protocols, positioning, and segmentation. This study systematically investigates the robustness of radiomics-based machine learning models under distribution shifts across five MRI sequences. We evaluated how different acquisition protocols and segmentation strategies affect model reliability in terms of predictive power and uncertainty-awareness. Using a phantom of 16 fruits, we evaluated distribution shifts through: (1) protocol variations across T2-HASTE, T2-TSE, T2-MAP, T1-TSE, and T2-FLAIR sequences; (2) segmentation variations (full, partial, rotated); and (3) inter-observer variability. We trained XGBoost classifiers on 8 consistent robust features versus sequence-specific features, testing model performance under in-domain and out-of-domain conditions. Results demonstrate that models trained on protocol-invariant features maintain F1-scores >0.85 across distribution shifts, while models using all features showed 40% performance degradation under protocol changes. Dataset augmentation substantially improved the quality of uncertainty estimates and reduced the expected calibration error (ECE) by 35% without sacrificing accuracy. Temperature scaling provided minimal calibration benefits, confirming XGBoost's inherent reliability. Our findings reveal that protocol-aware feature selection and controlled phantom studies effectively predict model behavior under distribution shifts, providing a framework for developing robust radiomics models resilient to real-world protocol variations.