Converting Annotated Clinical Cases into Structured Case Report Forms

TL;DR

This paper introduces a semi-automatic method to convert existing annotated datasets into structured clinical case report forms, creating high-quality CRF datasets to improve slot filling in medical research, and evaluates the performance of large language models on this task.

Contribution

The authors develop a novel semi-automatic conversion methodology to generate CRF datasets from existing annotations, enabling better training and evaluation of CRF slot filling systems.

Findings

CRF slot filling achieves 59.7% in Italian and 67.3% in English with large language models.

Performance is lower on open-source models, indicating the task's difficulty.

The created dataset is publicly available for further research.

Abstract

Case Report Forms (CRFs) are largely used in medical research as they ensure accuracy, reliability, and validity of results in clinical studies. However, publicly available, wellannotated CRF datasets are scarce, limiting the development of CRF slot filling systems able to fill in a CRF from clinical notes. To mitigate the scarcity of CRF datasets, we propose to take advantage of available datasets annotated for information extraction tasks and to convert them into structured CRFs. We present a semi-automatic conversion methodology, which has been applied to the E3C dataset in two languages (English and Italian), resulting in a new, high-quality dataset for CRF slot filling. Through several experiments on the created dataset, we report that slot filling achieves 59.7% for Italian and 67.3% for English on a closed Large Language Models (zero-shot) and worse performances on three families of open-source models, showing that filling CRFs is challenging even for recent state-of-the-art LLMs. We release the datest at https://huggingface.co/collections/NLP-FBK/e3c-to-crf-67b9844065460cbe42f80166