Towards Interpretable Foundation Models for Retinal Fundus Images

TL;DR

This paper introduces Dual-IFM, an interpretable foundation model for retinal fundus images that combines local and global interpretability with competitive performance.

Contribution

The paper presents Dual-IFM, a novel interpretable foundation model for retinal images that offers faithful local explanations and global visualization capabilities.

Findings

Achieves performance comparable to state-of-the-art models with fewer parameters.

Provides faithful class evidence maps for individual image interpretability.

Enables visualization of the model's representation space for dataset-level insights.

Abstract

Foundation models are used to extract transferable representations from large amounts of unlabeled data, typically via self-supervised learning (SSL). However, many of these models rely on architectures that offer limited interpretability, which is a critical issue in high-stakes domains such as medical imaging. We propose Dual-IFM, a foundation model that is interpretable-by-design in two ways: First, it provides local interpretability for individual images through class evidence maps that are faithful to the decision-making process. Second, it provides global interpretability for entire datasets through a 2D projection layer that allows for direct visualization of the model's representation space. We trained our model on over 800,000 color fundus photography from various sources to learn generalizable, interpretable representations for different downstream tasks. Our results show that our model reaches a performance range similar to that of state-of-the-art foundation models with up to $16\times$ the number of parameters, while providing interpretable predictions on out-of-distribution data. Our results suggest that large-scale SSL pretraining paired with inherent interpretability can lead to robust representations for retinal imaging.