Using Large Language Models to Extract and Assess Mobility Documentation for Age-Friendly Health Systems

TL;DR

This study shows that large language models can accurately extract mobility-related information from patient charts, potentially improving efficiency in age-friendly health systems.

Contribution

The study introduces a novel application of LLMs for automated extraction of mobility documentation in geriatric care.

Findings

LLMs showed moderate to substantial agreement with geriatricians in extracting mobility documentation.

LLMs performed reliably on both structured and unstructured medical notes.

Automating chart reviews with LLMs could reduce documentation burden and improve compliance.

Abstract

The Age-Friendly Health System (AFHS) uses the geriatric 4Ms framework (mind, mobility, medications, matters most) to improve health outcomes in older adults. Achieving AFHS Level 2 recognition in outpatient settings requires documentation of annual 4Ms care, but current methods to assess documentation, manual chart reviews or template from electronic health records are time-consuming and prone to inconsistency and errors. This study evaluates whether Large Language Models (LLMs) can accurately extract mobility-related documentation from patient charts, comparing their performance to manual reviews conducted by two geriatricians. Mobility documentation includes seven key components, including falls, activities of daily living (ADLs), instrumental ADLs, fall related injuries, gait and balance assessments, assistive device, and home safety. First, two geriatricians independently reviewed 26 structured geriatric consult notes (with a 4M-structured template) and 78 unstructured general medicine notes and reached consensus on mobility documentation. The LLMs prompts were fine-tuned to extract the same information from the structured notes. Next, we tested the prompts on the same 26 structured notes pooled with the 78 unstructured general medicine notes. We then assessed consensus on mobility documentation in these 104 notes between the two geriatricians (Cohen’s kappa 0.82), and geriatrician vs. LLMs (Cohen’s kappa 0.65), respectively, indicating moderate to substantial agreement. Our findings show that LLMs demonstrated high agreement with geriatricians across both note types, suggesting that LLMs can reliably chart review for mobility documentation in AFHS. This automation has the potential to streamline chart reviews, reduce documentation burden, enhance documentation compliance, and support the broader adoption of AFHS.