Using Large Text-to-Image Models with Structured Prompts for Skin Disease Identification: A Case Study

TL;DR

This study explores using large text-to-image models with structured prompts derived from medical textbooks to improve automated skin disease diagnosis, especially for rare conditions with limited data, demonstrating promising localization and generalization capabilities.

Contribution

It introduces a novel prompt structuring approach for LTI models based on medical textbooks, enabling effective diagnosis with scarce data and potential bias mitigation.

Findings

Enhanced localization of infected regions.

Effective diagnosis with limited annotated data.

Potential for cross-domain generalization.

Abstract

This paper investigates the potential usage of large text-to-image (LTI) models for the automated diagnosis of a few skin conditions with rarity or a serious lack of annotated datasets. As the input to the LTI model, we provide the targeted instantiation of a generic but succinct prompt structure designed upon careful observations of the conditional narratives from the standard medical textbooks. In this regard, we pave the path to utilizing accessible textbook descriptions for automated diagnosis of conditions with data scarcity through the lens of LTI models. Experiments show the efficacy of the proposed framework, including much better localization of the infected regions. Moreover, it has the immense possibility for generalization across the medical sub-domains, not only to mitigate the data scarcity issue but also to debias automated diagnostics from the all-pervasive racial biases.