UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling

TL;DR

UrFound introduces a universal retinal foundation model that leverages multimodal images and domain knowledge through a knowledge-guided masked modeling strategy, achieving superior generalization and data efficiency across ophthalmic tasks.

Contribution

The paper presents UrFound, a novel retinal foundation model that integrates multimodal retinal images and expert annotations using a knowledge-guided masked modeling approach for improved generalization.

Findings

Outperforms state-of-the-art models on 8 retinal datasets

Demonstrates strong generalization and data efficiency

Trained on approximately 180k images, surpassing models trained on larger datasets

Abstract

Retinal foundation models aim to learn generalizable representations from diverse retinal images, facilitating label-efficient model adaptation across various ophthalmic tasks. Despite their success, current retinal foundation models are generally restricted to a single imaging modality, such as Color Fundus Photography (CFP) or Optical Coherence Tomography (OCT), limiting their versatility. Moreover, these models may struggle to fully leverage expert annotations and overlook the valuable domain knowledge essential for domain-specific representation learning. To overcome these limitations, we introduce UrFound, a retinal foundation model designed to learn universal representations from both multimodal retinal images and domain knowledge. UrFound is equipped with a modality-agnostic image encoder and accepts either CFP or OCT images as inputs. To integrate domain knowledge into representation learning, we encode expert annotation in text supervision and propose a knowledge-guided masked modeling strategy for model pre-training. It involves reconstructing randomly masked patches of retinal images while predicting masked text tokens conditioned on the corresponding retinal image. This approach aligns multimodal images and textual expert annotations within a unified latent space, facilitating generalizable and domain-specific representation learning. Experimental results demonstrate that UrFound exhibits strong generalization ability and data efficiency when adapting to various tasks in retinal image analysis. By training on ~180k retinal images, UrFound significantly outperforms the state-of-the-art retinal foundation model trained on up to 1.6 million unlabelled images across 8 public retinal datasets. Our code and data are available at https://github.com/yukkai/UrFound.