Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection

TL;DR

This paper introduces StructuralGLIP, a novel approach that enhances zero-shot medical detection by encoding prompts into a knowledge bank for more precise image-text alignment, significantly improving detection accuracy.

Contribution

The paper proposes a new method that encodes prompts into a layered knowledge bank, enabling fine-grained, context-aware alignment between images and disease descriptions for zero-shot detection.

Findings

Achieves +4.1% AP improvement over state-of-the-art in seven benchmarks.

Improves fine-tuned models by +3.2% AP on endoscopy datasets.

Demonstrates effective structural representation for medical image-text alignment.

Abstract

Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase. However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance. Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets.