Data Alignment for Zero-Shot Concept Generation in Dermatology AI

TL;DR

This paper proposes a method to improve zero-shot dermatology image classification by aligning medical captions with natural language models, enhancing the performance of foundation models like CLIP.

Contribution

It introduces a novel approach of using fine-tuned LLMs to generate better-aligned captions for medical images, improving zero-shot classification accuracy.

Findings

Using GPT-3.5 generated captions improves classification performance.

Caption alignment with clinical lexicon enhances model trustworthiness.

Method leverages existing internet image-caption pairs for domain adaptation.

Abstract

AI in dermatology is evolving at a rapid pace but the major limitation to training trustworthy classifiers is the scarcity of data with ground-truth concept level labels, which are meta-labels semantically meaningful to humans. Foundation models like CLIP providing zero-shot capabilities can help alleviate this challenge by leveraging vast amounts of image-caption pairs available on the internet. CLIP can be fine-tuned using domain specific image-caption pairs to improve classification performance. However, CLIP's pre-training data is not well-aligned with the medical jargon that clinicians use to perform diagnoses. The development of large language models (LLMs) in recent years has led to the possibility of leveraging the expressive nature of these models to generate rich text. Our goal is to use these models to generate caption text that aligns well with both the clinical lexicon and with the natural human language used in CLIP's pre-training data. Starting with captions used for images in PubMed articles, we extend them by passing the raw captions through an LLM fine-tuned on the field's several textbooks. We find that using captions generated by an expressive fine-tuned LLM like GPT-3.5 improves downstream zero-shot concept classification performance.