Effortless Vision-Language Model Specialization in Histopathology without Annotation

TL;DR

This paper presents an annotation-free method for adapting vision-language models to histopathology tasks by continued pretraining on domain-specific image-caption pairs, improving performance without manual labels.

Contribution

It introduces a novel annotation-free adaptation approach via continued pretraining on existing image-caption data, enhancing VLMs for histopathology applications.

Findings

Continued pretraining improves zero-shot and few-shot performance.

Larger training sizes match few-shot results without manual labels.

The method is task-agnostic and effective across multiple tasks.

Abstract

Recent advances in Vision-Language Models (VLMs) in histopathology, such as CONCH and QuiltNet, have demonstrated impressive zero-shot classification capabilities across various tasks. However, their general-purpose design may lead to suboptimal performance in specific downstream applications. While supervised fine-tuning methods address this issue, they require manually labeled samples for adaptation. This paper investigates annotation-free adaptation of VLMs through continued pretraining on domain- and task-relevant image-caption pairs extracted from existing databases. Our experiments on two VLMs, CONCH and QuiltNet, across three downstream tasks reveal that these pairs substantially enhance both zero-shot and few-shot performance. Notably, with larger training sizes, continued pretraining matches the performance of few-shot methods while eliminating manual labeling. Its effectiveness, task-agnostic design, and annotation-free workflow make it a promising pathway for adapting VLMs to new histopathology tasks. Code is available at https://github.com/DeepMicroscopy/Annotation-free-VLM-specialization.