Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?

TL;DR

This paper reviews sharpness-based optimizers, especially SAM, and evaluates their effectiveness in improving deep learning model generalization on medical breast ultrasound images, highlighting SAM's unique benefits in this domain.

Contribution

The study provides a comprehensive review of recent sharpness-based optimization methods and evaluates their performance on medical imaging, revealing SAM's consistent benefits for medical image analysis.

Findings

SAM improves generalization in medical deep learning models

Adaptive SAM benefits CNNs but not vision transformers

Other sharpness-based optimizers show inconsistent results

Abstract

Effective clinical deployment of deep learning models in healthcare demands high generalization performance to ensure accurate diagnosis and treatment planning. In recent years, significant research has focused on improving the generalization of deep learning models by regularizing the sharpness of the loss landscape. Among the optimization approaches that explicitly minimize sharpness, Sharpness-Aware Minimization (SAM) has shown potential in enhancing generalization performance on general domain image datasets. This success has led to the development of several advanced sharpness-based algorithms aimed at addressing the limitations of SAM, such as Adaptive SAM, surrogate-Gap SAM, Weighted SAM, and Curvature Regularized SAM. These sharpness-based optimizers have shown improvements in model generalization compared to conventional stochastic gradient descent optimizers and their variants on general domain image datasets, but they have not been thoroughly evaluated on medical images. This work provides a review of recent sharpness-based methods for improving the generalization of deep learning networks and evaluates the methods performance on medical breast ultrasound images. Our findings indicate that the initial SAM method successfully enhances the generalization of various deep learning models. While Adaptive SAM improves generalization of convolutional neural networks, it fails to do so for vision transformers. Other sharpness-based optimizers, however, do not demonstrate consistent results. The results reveal that, contrary to findings in the non-medical domain, SAM is the only recommended sharpness-based optimizer that consistently improves generalization in medical image analysis, and further research is necessary to refine the variants of SAM to enhance generalization performance in this field