FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

TL;DR

This paper introduces FHIRPath-QA, a new dataset and benchmark for patient-specific question answering over electronic health records using FHIRPath queries, demonstrating that query synthesis improves efficiency and accuracy in clinical QA tasks.

Contribution

The work presents the first open dataset and benchmark for FHIRPath-based clinical question answering, proposing a text-to-FHIRPath paradigm that enhances efficiency and accuracy over traditional retrieval methods.

Findings

Supervised fine-tuning improves query synthesis accuracy from 27% to 79%.

Text-to-FHIRPath reduces token usage by 391x compared to retrieval-based prompting.

LLMs achieve up to 42% accuracy on the task, indicating room for improvement.

Abstract

Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. This work introduces FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. A text-to-FHIRPath QA paradigm is proposed that shifts reasoning from free-text generation to FHIRPath query synthesis. For o4-mini, this reduced average token usage by 391x relative to retrieval-first prompting (629,829 vs 1,609 tokens per question) and lowered failure rates from 0.36 to 0.09 on clinician-phrased questions. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Empirically, the evaluated LLMs achieve at most 42% accuracy, highlighting the challenge of the task, but benefit strongly from supervised fine-tuning, with query synthesis accuracy improving from 27% to 79% for 4o-mini. These results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and the FHIRPath-QA dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code can be accessed at: https://github.com/mooshifrew/fhirpath-qa.