Glo-VLMs: Leveraging Vision-Language Models for Fine-Grained Diseased Glomerulus Classification

TL;DR

This paper introduces Glo-VLMs, a framework that adapts vision-language models for fine-grained glomerular disease classification in pathology, demonstrating high accuracy with limited labeled data.

Contribution

It systematically explores adaptation strategies of large pretrained VLMs for renal pathology classification under data-constrained conditions.

Findings

Achieved 0.7416 accuracy with 8 shots per class.

Demonstrated effective joint image-text representation learning.

Showed potential of foundation models in specialized clinical tasks.

Abstract

Vision-language models (VLMs) have shown considerable potential in digital pathology, yet their effectiveness remains limited for fine-grained, disease-specific classification tasks such as distinguishing between glomerular subtypes. The subtle morphological variations among these subtypes, combined with the difficulty of aligning visual patterns with precise clinical terminology, make automated diagnosis in renal pathology particularly challenging. In this work, we explore how large pretrained VLMs can be effectively adapted to perform fine-grained glomerular classification, even in scenarios where only a small number of labeled examples are available. In this work, we introduce Glo-VLMs, a systematic framework designed to explore the adaptation of VLMs to fine-grained glomerular classification in data-constrained settings. Our approach leverages curated pathology images alongside clinical text prompts to facilitate joint image-text representation learning for nuanced renal pathology subtypes. By assessing various VLMs architectures and adaptation strategies under a few-shot learning paradigm, we explore how both the choice of method and the amount of labeled data impact model performance in clinically relevant scenarios. To ensure a fair comparison, we evaluate all models using standardized multi-class metrics, aiming to clarify the practical requirements and potential of large pretrained models for specialized clinical research applications. As a result, fine-tuning the VLMs achieved 0.7416 accuracy, 0.9045 macro-AUC, and 0.5277 F1-score with only 8 shots per class, demonstrating that even with highly limited supervision, foundation models can be effectively adapted for fine-grained medical image classification.