Generalizability vs. Robustness: Adversarial Examples for Medical Imaging

TL;DR

This paper introduces a novel evaluation method using adversarial examples to assess the robustness of medical imaging models against noise, outliers, and ambiguous data, revealing significant differences in model perception.

Contribution

It proposes a new robustness evaluation framework for medical imaging models using adversarial examples, highlighting disparities not captured by traditional generalization metrics.

Findings

Models with similar generalization performance can differ greatly in robustness.

Adversarial evaluation uncovers vulnerabilities in state-of-the-art medical imaging models.

Robustness varies significantly across different architectures like Inception and UNet.

Abstract

In this paper, for the first time, we propose an evaluation method for deep learning models that assesses the performance of a model not only in an unseen test scenario, but also in extreme cases of noise, outliers and ambiguous input data. To this end, we utilize adversarial examples, images that fool machine learning models, while looking imperceptibly different from original data, as a measure to evaluate the robustness of a variety of medical imaging models. Through extensive experiments on skin lesion classification and whole brain segmentation with state-of-the-art networks such as Inception and UNet, we show that models that achieve comparable performance regarding generalizability may have significant variations in their perception of the underlying data manifold, leading to an extensive performance gap in their robustness.