Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department

TL;DR

This paper introduces a new annotated dataset of Italian emergency department clinical notes for automatic case report form filling, demonstrating that large language models can perform this task in a zero-shot setting with bias considerations.

Contribution

It provides the first annotated dataset for Italian clinical notes targeting CRF filling and evaluates LLM performance, highlighting challenges and biases in zero-shot approaches.

Findings

LLMs can perform CRF filling in Italian clinical notes without prior training.

Biases in LLM responses, such as overusing 'unknown', affect accuracy.

Zero-shot LLM approaches show promise but require bias correction.

Abstract

Case Report Forms (CRFs) collect data about patients and are at the core of well-established practices to conduct research in clinical settings. With the recent progress of language technologies, there is an increasing interest in automatic CRF-filling from clinical notes, mostly based on the use of Large Language Models (LLMs). However, there is a general scarcity of annotated CRF data, both for training and testing LLMs, which limits the progress on this task. As a step in the direction of providing such data, we present a new dataset of clinical notes from an Italian Emergency Department annotated with respect to a pre-defined CRF containing 134 items to be filled. We provide an analysis of the data, define the CRF-filling task and metric for its evaluation, and report on pilot experiments where we use an open-source state-of-the-art LLM to automatically execute the task. Results of the case-study show that (i) CRF-filling from real clinical notes in Italian can be approached in a zero-shot setting; (ii) LLMs' results are affected by biases (e.g., a cautious behaviour favours "unknown" answers), which need to be corrected.