Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM

TL;DR

This paper presents a novel zero-shot medical image classification method that generates customized prompts using domain-specific expert knowledge, improving rare event detection without additional training.

Contribution

It introduces a new approach to generate contextually relevant prompts for large language models, enhancing zero-shot classification of rare medical events.

Findings

Outperforms state-of-the-art zero-shot methods in rare event medical image classification

Uses domain-specific expert knowledge to generate effective prompts

Achieves high accuracy without additional training data

Abstract

Rare events, due to their infrequent occurrences, do not have much data, and hence deep learning techniques fail in estimating the distribution for such data. Open-vocabulary models represent an innovative approach to image classification. Unlike traditional models, these models classify images into any set of categories specified with natural language prompts during inference. These prompts usually comprise manually crafted templates (e.g., 'a photo of a {}') that are filled in with the names of each category. This paper introduces a simple yet effective method for generating highly accurate and contextually descriptive prompts containing discriminative characteristics. Rare event detection, especially in medicine, is more challenging due to low inter-class and high intra-class variability. To address these, we propose a novel approach that uses domain-specific expert knowledge on rare events to generate customized and contextually relevant prompts, which are then used by large language models for image classification. Our zero-shot, privacy-preserving method enhances rare event classification without additional training, outperforming state-of-the-art techniques.