# Assessment of the Modified Rankin Scale in Electronic Health Records With a Fine-Tuned Large Language Model: Development and Internal Validation

## TL;DR

This study shows that a large language model can accurately determine stroke recovery scores from electronic health records, though it struggles with some score distinctions.

## Contribution

A fine-tuned large language model is developed to automate the classification of modified Rankin Scale scores from EHRs.

## Key findings

- The multiclass model achieved 77% accuracy and a weighted Cohen κ of 0.92 for all seven mRS scores.
- The binary model achieved 92% accuracy and a Cohen κ of 0.84 for classifying functional independence.
- The model performed best for mRS score 4 (90% accuracy) and worst for mRS score 2 (28% accuracy).

## Abstract

The modified Rankin scale (mRS) is an important metric in stroke research, often used as a primary outcome in clinical trials and observational studies. The mRS can be assessed retrospectively from electronic health records (EHRs), but this process is labor-intensive and prone to interrater variability. Large language models (LLMs) have demonstrated potential in automating text classification.

We aimed to create a fine-tuned LLM that can analyze EHR text and classify mRS scores for clinical and research applications.

We performed a retrospective cohort study of patients admitted to a specialist stroke neurology service at a large academic hospital system between August 2020 and June 2023. Each patient’s medical record was reviewed at two time points: (1) at hospital discharge and (2) approximately 90 days post discharge. Two independent researchers assigned an mRS score at each time point. Two separate models were trained on EHR passages with corresponding mRS scores as labeled outcomes: (1) a multiclass model to classify all seven mRS scores and (2) a binary model to classify functional independence (mRS scores 0‐2) versus non-independence (mRS scores 3‐6). Four-fold cross-validation was conducted using accuracy and the Cohen κ as model performance metrics.

A total of 2290 EHR passages with corresponding mRS scores were included in model training. The multiclass model—considering all seven scores of the mRS—attained an accuracy of 77% and a weighted Cohen κ of 0.92. Class-specific accuracy was the highest for mRS score 4 (90%) and the lowest for mRS score 2 (28%). The binary model—considering only functional independence versus non-independence—attained an accuracy of 92% and a Cohen κ of 0.84.

Our findings demonstrate that LLMs can be successfully trained to determine mRS scores through EHR text analysis; however, improving discrimination between intermediate scores is required.