Pseudo-Prompt Generating in Pre-trained Vision-Language Models for Multi-Label Medical Image Classification

TL;DR

This paper introduces PsPG, a novel method that automatically generates class-specific prompts for pre-trained vision-language models to improve multi-label medical image classification, especially for unseen categories.

Contribution

The paper proposes PsPG, a text generation-inspired prompt generation approach that enhances zero-shot multi-label classification in medical imaging by leveraging multi-modal feature priors.

Findings

PsPG outperforms existing prompt learning methods on chest radiograph datasets.

The approach effectively generalizes to unseen categories in multi-label classification.

Automated pseudo-prompt generation reduces reliance on manual prompt design.

Abstract

The task of medical image recognition is notably complicated by the presence of varied and multiple pathological indications, presenting a unique challenge in multi-label classification with unseen labels. This complexity underlines the need for computer-aided diagnosis methods employing multi-label zero-shot learning. Recent advancements in pre-trained vision-language models (VLMs) have showcased notable zero-shot classification abilities on medical images. However, these methods have limitations on leveraging extensive pre-trained knowledge from broader image datasets, and often depend on manual prompt construction by expert radiologists. By automating the process of prompt tuning, prompt learning techniques have emerged as an efficient way to adapt VLMs to downstream tasks. Yet, existing CoOp-based strategies fall short in performing class-specific prompts on unseen categories, limiting generalizability in fine-grained scenarios. To overcome these constraints, we introduce a novel prompt generation approach inspirited by text generation in natural language processing (NLP). Our method, named Pseudo-Prompt Generating (PsPG), capitalizes on the priori knowledge of multi-modal features. Featuring a RNN-based decoder, PsPG autoregressively generates class-tailored embedding vectors, i.e., pseudo-prompts. Comparative evaluations on various multi-label chest radiograph datasets affirm the superiority of our approach against leading medical vision-language and multi-label prompt learning methods. The source code is available at https://github.com/fallingnight/PsPG