Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale

TL;DR

This study systematically evaluates the quality of LLM-generated synthetic clinical notes at a large scale, assessing their information preservation, utility, and errors, with insights on rephrasing strategies and task-specific benefits.

Contribution

It provides a comprehensive analysis of synthetic clinical notes, highlighting how rephrasing methods affect information retention and error types, and demonstrates their utility in augmenting rare ICD code classification.

Findings

Synthetic notes preserve core clinical info and utility despite linguistic changes.

Rephrasing by chunks reduces detail loss but lowers factual accuracy.

Synthetic notes can augment training for rare ICD codes.

Abstract

Large language models (LLMs) can generate or synthesize clinical text for a wide range of applications, from improving clinical documentation to augmenting clinical text analytics. Yet evaluations typically focus on a narrow aspect -- such as similarity or utility comparisons -- even though these aspects are complementary and best viewed in parallel. In this study, we aim to conduct a systematic evaluation of LLM-generated clinical text, which includes intrinsic, extrinsic, and factuality evaluations of synthetic clinical notes rephrased from MIMIC databases at million-note scale. Our analysis demonstrates that synthetic notes preserve core clinical information and predictive utility for coarse-grained tasks despite substantial linguistic changes, but lose fine-grained details for task like ICD coding. We show this loss of detail can be substantially mitigated by rephrasing notes by chunks rather than by the whole note, but at the cost of reduced factual precision under incomplete context. Through fact-checking and error analysis, we further find that synthesis errors are dominated by misinterpretation of clinical context, alongside temporal confusion, measurement errors, and fabricated claims. Finally, we show that the synthetic notes -- despite their task-agnostic nature -- can effectively augment task-specific training for rare ICD codes.