Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

TL;DR

This study demonstrates that open-source large language models can accurately retrieve patient-specific information from Finnish electronic health records through a locally deployable question answering framework, emphasizing clinical validation.

Contribution

The paper introduces a fully offline, open-source LLM-based framework for clinical question answering from EHRs, with benchmarking on Finnish clinical data and analysis of deployment considerations.

Findings

Llama-3.1-70B achieved 95.3% accuracy in free-text generation.

Models maintained performance with low-precision quantization.

Clinically significant errors occurred in 2.9% of outputs.

Abstract

Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-language queries, while highlighting the need for validation and human oversight in clinical deployment.