Boosting Vision-Language Models for Histopathology Classification: Predict all at once

TL;DR

This paper introduces a transductive approach that enhances vision-language models for histopathology classification by leveraging patch relationships and text-based predictions, achieving significant accuracy improvements without additional labels.

Contribution

It extends existing VLMs with a transductive method that uses affinity among patches, improving zero-shot classification in histopathology without extra labels.

Findings

Significant accuracy improvements over inductive zero-shot classification

Efficient processing of 100,000 patches in seconds

Validated on four histopathology datasets with five VLMs

Abstract

The development of vision-language models (VLMs) for histo-pathology has shown promising new usages and zero-shot performances. However, current approaches, which decompose large slides into smaller patches, focus solely on inductive classification, i.e., prediction for each patch is made independently of the other patches in the target test data. We extend the capability of these large models by introducing a transductive approach. By using text-based predictions and affinity relationships among patches, our approach leverages the strong zero-shot capabilities of these new VLMs without any additional labels. Our experiments cover four histopathology datasets and five different VLMs. Operating solely in the embedding space (i.e., in a black-box setting), our approach is highly efficient, processing $10^5$ patches in just a few seconds, and shows significant accuracy improvements over inductive zero-shot classification. Code available at https://github.com/FereshteShakeri/Histo-TransCLIP.