Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AI

TL;DR

This paper explores methods for generating synthetic clinical notes using LLMs to improve multimodal dermatology AI models, enhancing classification accuracy and enabling cross-modal retrieval despite limited real data.

Contribution

It introduces strategies for prompt design and metadata inclusion to synthesize clinical notes, improving model robustness and cross-modal capabilities in medical AI.

Findings

Synthetic notes improve classification accuracy under domain shift

Enhanced cross-modal retrieval capabilities are demonstrated

Synthetic data mitigates data scarcity issues in dermatology AI

Abstract

Multimodal (MM) learning is emerging as a promising paradigm in biomedical artificial intelligence (AI) applications, integrating complementary modality, which highlight different aspects of patient health. The scarcity of large heterogeneous biomedical MM data has restrained the development of robust models for medical AI applications. In the dermatology domain, for instance, skin lesion datasets typically include only images linked to minimal metadata describing the condition, thereby limiting the benefits of MM data integration for reliable and generalizable predictions. Recent advances in Large Language Models (LLMs) enable the synthesis of textual description of image findings, potentially allowing the combination of image and text representations. However, LLMs are not specifically trained for use in the medical domain, and their naive inclusion has raised concerns about the risk of hallucinations in clinically relevant contexts. This work investigates strategies for generating synthetic textual clinical notes, in terms of prompt design and medical metadata inclusion, and evaluates their impact on MM architectures toward enhancing performance in classification and cross-modal retrieval tasks. Experiments across several heterogeneous dermatology datasets demonstrate that synthetic clinical notes not only enhance classification performance, particularly under domain shift, but also unlock cross-modal retrieval capabilities, a downstream task that is not explicitly optimized during training.