Exploring the Transfer Learning Capabilities of CLIP in Domain Generalization for Diabetic Retinopathy

TL;DR

This paper evaluates CLIP's transfer learning ability for domain generalization in diabetic retinopathy classification and introduces a novel multi-modal fine-tuning method that improves performance across domains.

Contribution

It investigates CLIP's potential for cross-domain diabetic retinopathy classification and proposes CoOpLVT, a new fine-tuning strategy that enhances generalization.

Findings

CoOpLVT increases F1-score by 1.8% over baseline

CLIP shows promising transfer learning capabilities for DR DG

Proposed method improves cross-domain classification performance

Abstract

Diabetic Retinopathy (DR), a leading cause of vision impairment, requires early detection and treatment. Developing robust AI models for DR classification holds substantial potential, but a key challenge is ensuring their generalization in unfamiliar domains with varying data distributions. To address this, our paper investigates cross-domain generalization, also known as domain generalization (DG), within the context of DR classification. DG, a challenging problem in the medical domain, is complicated by the difficulty of gathering labeled data across different domains, such as patient demographics and disease stages. Some recent studies have shown the effectiveness of using CLIP to handle the DG problem in natural images. In this study, we investigate CLIP's transfer learning capabilities and its potential for cross-domain generalization in diabetic retinopathy (DR) classification. We carry out comprehensive experiments to assess the efficacy and potential of CLIP in addressing DG for DR classification. Further, we introduce a multi-modal fine-tuning strategy named Context Optimization with Learnable Visual Tokens (CoOpLVT), which enhances context optimization by conditioning on visual features. Our findings demonstrate that the proposed method increases the F1-score by 1.8% over the baseline, thus underlining its promise for effective DG in DR classification. Our code is publicly available at https://github.com/Sanoojan/CLIP-DRDG.