Interpretable 2D Vision Models for 3D Medical Images

TL;DR

This paper introduces an interpretable method for applying 2D neural networks to 3D medical images by using attention pooling to weight slices, enabling both effective performance and interpretability.

Contribution

It proposes a simple adaptation of 2D networks with attention pooling for 3D images, providing built-in interpretability and competitive results on medical imaging benchmarks.

Findings

Performs on par with existing 3D methods on MedMNIST and real-world datasets.

Provides direct interpretability through attention weights.

Achieves comparable accuracy with enhanced transparency.

Abstract

Training Artificial Intelligence (AI) models on 3D images presents unique challenges compared to the 2D case: Firstly, the demand for computational resources is significantly higher, and secondly, the availability of large datasets for pre-training is often limited, impeding training success. This study proposes a simple approach of adapting 2D networks with an intermediate feature representation for processing 3D images. Our method employs attention pooling to learn to assign each slice an importance weight and, by that, obtain a weighted average of all 2D slices. These weights directly quantify the contribution of each slice to the contribution and thus make the model prediction inspectable. We show on all 3D MedMNIST datasets as benchmark and two real-world datasets consisting of several hundred high-resolution CT or MRI scans that our approach performs on par with existing methods. Furthermore, we compare the in-built interpretability of our approach to HiResCam, a state-of-the-art retrospective interpretability approach.