Learning to Adapt Foundation Model DINOv2 for Capsule Endoscopy Diagnosis

TL;DR

This paper presents a low-rank adaptation method to fine-tune foundation models like DINOv2 for capsule endoscopy diagnosis, achieving high accuracy on public datasets with minimal training adjustments.

Contribution

Introduces a LoRA-based adaptation approach for foundation models, enabling effective and efficient customization for medical imaging tasks like capsule endoscopy diagnosis.

Findings

Achieved 97.75% accuracy on Kvasir-Capsule dataset.

Achieved 98.81% accuracy on Kvasirv2 dataset.

Demonstrated the effectiveness of LoRA for domain-specific model adaptation.

Abstract

Foundation models have become prominent in computer vision, achieving notable success in various tasks. However, their effectiveness largely depends on pre-training with extensive datasets. Applying foundation models directly to small datasets of capsule endoscopy images from scratch is challenging. Pre-training on broad, general vision datasets is crucial for successfully fine-tuning our model for specific tasks. In this work, we introduce a simplified approach called Adapt foundation models with a low-rank adaptation (LoRA) technique for easier customization. Our method, inspired by the DINOv2 foundation model, applies low-rank adaptation learning to tailor foundation models for capsule endoscopy diagnosis effectively. Unlike traditional fine-tuning methods, our strategy includes LoRA layers designed to absorb specific surgical domain knowledge. During the training process, we keep the main model (the backbone encoder) fixed and focus on optimizing the LoRA layers and the disease classification component. We tested our method on two publicly available datasets for capsule endoscopy disease classification. The results were impressive, with our model achieving 97.75% accuracy on the Kvasir-Capsule dataset and 98.81% on the Kvasirv2 dataset. Our solution demonstrates that foundation models can be adeptly adapted for capsule endoscopy diagnosis, highlighting that mere reliance on straightforward fine-tuning or pre-trained models from general computer vision tasks is inadequate for such specific applications.