Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy

TL;DR

This paper assesses the effectiveness of vision foundation models in improving pixel and object classification tasks in microscopy, comparing them to traditional methods across diverse datasets.

Contribution

It introduces a comprehensive benchmark for VFMs in microscopy and evaluates their performance in pixel and object classification tasks.

Findings

VFMs outperform traditional hand-crafted features in microscopy classification tasks.

Evaluation of multiple VFMs shows consistent improvements across datasets.

The study provides a benchmark for future research in microscopy vision models.

Abstract

Deep learning underlies most modern approaches and tools in computer vision, including biomedical imaging. However, for interactive semantic segmentation (often called pixel classification in this context) and interactive object-level classification (object classification), feature-based shallow learning remains widely used. This is due to the diversity of data in this domain, the lack of large pretraining datasets, and the need for computational and label efficiency. In contrast, state-of-the-art tools for many other vision tasks in microscopy - most notably cellular instance segmentation - already rely on deep learning and have recently benefited substantially from vision foundation models (VFMs), particularly SAM. Here, we investigate whether VFMs can also improve pixel and object classification compared to current approaches. To this end, we evaluate several VFMs, including general-purpose models (SAM, SAM2, DINOv3) and domain-specific ones ($\mu$SAM, PathoSAM), in combination with shallow learning and attentive probing on five diverse and challenging datasets. Our results demonstrate consistent improvements over hand-crafted features and provide a clear pathway toward practical improvements. Furthermore, our study establishes a benchmark for VFMs in microscopy and informs future developments in this area.