In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy

TL;DR

This paper explores how in-context learning enables vision-language models to perform few-shot cell detection in microscopy images, introducing a new benchmark and a hybrid detection pipeline to improve performance.

Contribution

It introduces the Micro-OD benchmark for microscopy, evaluates VLMs for few-shot detection, and proposes a hybrid pipeline combining detection and classification for enhanced results.

Findings

Few-shot learning improves detection performance over zero-shot.

Models with reasoning tokens excel in localization tasks.

Marginal gains after six shots indicate early saturation.

Abstract

Foundation vision-language models (VLMs) excel on natural images, but their utility for biomedical microscopy remains underexplored. In this paper, we investigate how in-context learning enables state-of-the-art VLMs to perform few-shot object detection when large annotated datasets are unavailable, as is often the case with microscopic images. We introduce the Micro-OD benchmark, a curated collection of 252 images specifically curated for in-context learning, with bounding-box annotations spanning 11 cell types across four sources, including two in-lab expert-annotated sets. We systematically evaluate eight VLMs under few-shot conditions and compare variants with and without implicit test-time reasoning tokens. We further implement a hybrid Few-Shot Object Detection (FSOD) pipeline that combines a detection head with a VLM-based few-shot classifier, which enhances the few-shot performance of recent VLMs on our benchmark. Across datasets, we observe that zero-shot performance is weak due to the domain gap; however, few-shot support consistently improves detection, with marginal gains achieved after six shots. We observe that models with reasoning tokens are more effective for end-to-end localization, whereas simpler variants are more suitable for classifying pre-localized crops. Our results highlight in-context adaptation as a practical path for microscopy, and our benchmark provides a reproducible testbed for advancing open-vocabulary detection in biomedical imaging.