Zero-shot Nuclei Detection via Visual-Language Pre-trained Models

TL;DR

This paper explores the use of large-scale visual-language pre-trained models for zero-shot nuclei detection in medical images, introducing an automatic prompts design and self-training framework that significantly improves detection performance.

Contribution

It presents a novel automatic prompts design pipeline and a self-training framework for zero-shot nuclei detection using VLPMs, demonstrating their potential in medical imaging.

Findings

Achieved superior zero-shot nuclei detection performance compared to other methods.

Demonstrated the potential of natural image-text pre-trained models for medical image analysis.

Established an effective iterative refinement process for label-free detection.

Abstract

Large-scale visual-language pre-trained models (VLPM) have proven their excellent performance in downstream object detection for natural scenes. However, zero-shot nuclei detection on H\&E images via VLPMs remains underexplored. The large gap between medical images and the web-originated text-image pairs used for pre-training makes it a challenging task. In this paper, we attempt to explore the potential of the object-level VLPM, Grounded Language-Image Pre-training (GLIP) model, for zero-shot nuclei detection. Concretely, an automatic prompts design pipeline is devised based on the association binding trait of VLPM and the image-to-text VLPM BLIP, avoiding empirical manual prompts engineering. We further establish a self-training framework, using the automatically designed prompts to generate the preliminary results as pseudo labels from GLIP and refine the predicted boxes in an iterative manner. Our method achieves a remarkable performance for label-free nuclei detection, surpassing other comparison methods. Foremost, our work demonstrates that the VLPM pre-trained on natural image-text pairs exhibits astonishing potential for downstream tasks in the medical field as well. Code will be released at https://github.com/wuyongjianCODE/VLPMNuD.