Impact of Large Language Model Assistance on Patients Reading Clinical Notes: A Mixed-Methods Study

TL;DR

This study evaluates a new LLM-based tool designed to improve patient understanding of clinical notes, demonstrating increased comprehension and capturing patient and clinician perspectives on AI-assisted note interpretation.

Contribution

Introduces a novel LLM-powered patient-facing tool that enhances clinical note readability and understanding, with mixed-methods evaluation including patient and clinician insights.

Findings

Augmentations significantly improved patient understanding scores.

Patients expressed positive sentiments towards AI assistance.

Clinicians identified specific error modes of the model.

Abstract

Large language models (LLMs) have immense potential to make information more accessible, particularly in medicine, where complex medical jargon can hinder patient comprehension of clinical notes. We developed a patient-facing tool using LLMs to make clinical notes more readable by simplifying, extracting information from, and adding context to the notes. We piloted the tool with clinical notes donated by patients with a history of breast cancer and synthetic notes from a clinician. Participants (N=200, healthy, female-identifying patients) were randomly assigned three clinical notes in our tool with varying levels of augmentations and answered quantitative and qualitative questions evaluating their understanding of follow-up actions. Augmentations significantly increased their quantitative understanding scores. In-depth interviews were conducted with participants (N=7, patients with a history of breast cancer), revealing both positive sentiments about the augmentations and concerns about AI. We also performed a qualitative clinician-driven analysis of the model's error modes.