LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications

TL;DR

This paper evaluates the effectiveness of large language models, especially GPT-4, in identifying relevant sections in electronic health records, showing strong performance in controlled settings but challenges in real-world applications.

Contribution

It demonstrates GPT-4's superior zero and few-shot performance in section identification for EHRs and highlights the gap in real-world effectiveness, proposing new benchmarks.

Findings

GPT-4 outperforms state-of-the-art methods in controlled settings.

GPT-4 struggles on real-world, annotated datasets.

Zero and few-shot capabilities are promising but not sufficient for practical use.

Abstract

Electronic health records (EHR) even though a boon for healthcare practitioners, are growing convoluted and longer every day. Sifting around these lengthy EHRs is taxing and becomes a cumbersome part of physician-patient interaction. Several approaches have been proposed to help alleviate this prevalent issue either via summarization or sectioning, however, only a few approaches have truly been helpful in the past. With the rise of automated methods, machine learning (ML) has shown promise in solving the task of identifying relevant sections in EHR. However, most ML methods rely on labeled data which is difficult to get in healthcare. Large language models (LLMs) on the other hand, have performed impressive feats in natural language processing (NLP), that too in a zero-shot manner, i.e. without any labeled data. To that end, we propose using LLMs to identify relevant section headers. We find that GPT-4 can effectively solve the task on both zero and few-shot settings as well as segment dramatically better than state-of-the-art methods. Additionally, we also annotate a much harder real world dataset and find that GPT-4 struggles to perform well, alluding to further research and harder benchmarks.