PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification

TL;DR

This paper introduces PM2, a multi-modal prompting paradigm for few-shot medical image classification that leverages both image and text prompts, utilizing rich visual statistics and multi-modal foundation models to improve performance.

Contribution

The paper proposes a novel multi-modal prompting framework, PM2, combining image and text prompts with advanced feature aggregation for enhanced few-shot medical image classification.

Findings

PM2 outperforms existing methods on three medical datasets.

Using combined classification heads improves accuracy.

Empirical investigation of five prompt schemes demonstrates robustness.

Abstract

Few-shot learning has been successfully applied to medical image classification as only very few medical examples are available for training. Due to the challenging problem of limited number of annotated medical images, image representations should not be solely derived from a single image modality which is insufficient for characterizing concept classes. In this paper, we propose a new prompting multi-modal model paradigm on medical image classification based on multi-modal foundation models, called PM2. Besides image modality,PM2 introduces another supplementary text input, known as prompt, to further describe corresponding image or concept classes and facilitate few-shot learning across diverse modalities. To better explore the potential of prompt engineering, we empirically investigate five distinct prompt schemes under the new paradigm. Furthermore, linear probing in multi-modal models acts as a linear classification head taking as input only class token, which ignores completely merits of rich statistics inherent in high-level visual tokens. Thus, we alternatively perform a linear classification on feature distribution of visual tokens and class token simultaneously. To effectively mine such rich statistics, a global covariance pooling with efficient matrix power normalization is used to aggregate visual tokens. Then we study and combine two classification heads. One is shared for class token of image from vision encoder and prompt representation encoded by text encoder. The other is to classification on feature distribution of visual tokens from vision encoder. Extensive experiments on three medical datasets show that our PM2 significantly outperforms counterparts regardless of prompt schemes and achieves state-of-the-art performance.