# Leveraging large language models to populate structured clinical case report forms from unstructured medical notes in radiation oncology

## TL;DR

This paper shows that large language models can efficiently extract and structure clinical data from unstructured medical notes in radiation oncology, achieving high accuracy.

## Contribution

The study demonstrates the first application of LLMs for structuring non-English clinical notes in radiation oncology with minimal manual input.

## Key findings

- LLMs processed medical notes in 16 seconds per note with 83.6% and 83.8% accuracy on development and testing datasets.
- Model disagreement occurred in 8.1% of development and 8.6% of testing cases.
- Manual review revealed 7.5% of routine test data had mismatches, suggesting ground truth inaccuracies.

## Abstract

•Large language models (LLMs) can automatically extract and structure data from unstructured medical notes with an average time of 16 s per note.•The LLM achieved matching accuracies of 83.6% and 83.8% on the development and testing datasets, respectively.•In-depth analysis revealed model disagreement with specific values in 8.1% (development) and 8.6% (testing) of cases.•Manual review found ∼7.5% of routine test data mismatched reviewed values, indicating inaccuracies in the routine ground truth.

Large language models (LLMs) can automatically extract and structure data from unstructured medical notes with an average time of 16 s per note.

The LLM achieved matching accuracies of 83.6% and 83.8% on the development and testing datasets, respectively.

In-depth analysis revealed model disagreement with specific values in 8.1% (development) and 8.6% (testing) of cases.

Manual review found ∼7.5% of routine test data mismatched reviewed values, indicating inaccuracies in the routine ground truth.

Large language models (LLMs) have shown growing potential for clinical text processing, but their systematic application in radiation oncology—especially for non-English clinical documentation—remains underexplored. This study investigated whether pretrained LLMs can automatically extract, analyze, and structure radiotherapy-relevant information from routine unstructured medical notes, with the goal of supporting automated population of electronic case report forms (eCRFs).

This study examined prostate cancer patients treated with the MR-Linac, for whom ground truth data exist in the MOMENTUM database. A total of 100 patients were included, with 90 used for prompt development and 10 for independent testing. Medical notes were extracted, anonymized, and categorized by time points. The Llama-3.1-8b model was used, with prompts designed using chain-of-thought (CoT) logic with five in-context examples. The model output was post-processed, and extracted data was compared against ground truth.

Medical notes were successfully processed, with predicted values generated in an average time of 16 s per note. The LLM achieved matching accuracies of 83.6% and 83.8% on the development and testing datasets. Analysis revealed that the model disagreed with specific values in 8.1% of development dataset cases and 8.6% of testing dataset cases. An independent manual review before model evaluation showed approximately 7.5% of routinely collected test data did not match reviewed values, indicating inaccuracies in the routinely acquired ground truth.

This study demonstrated the effectiveness of LLMs in structuring clinical data from medical non-English notes, with high accuracy in extracting and categorizing information. While multi-institutional validation is needed, the results indicate a significant healthcare impact through efficient data management, processing notes in 16 s, and accurately populating CRFs with minimal staff involvement.