Toward Automated Cognitive Assessment in Parkinson's Disease Using Pretrained Language Models

TL;DR

This study evaluates NLP models for automatically extracting cognitive process categories from patient narratives in Parkinson's disease, aiming to support low-burden, longitudinal cognitive monitoring.

Contribution

It compares multiple advanced NLP models, including fine-tuned Llama and GPT-4, for their effectiveness in identifying complex cognitive categories from unstructured narratives.

Findings

Meta-Llama-3-8B-Instruct achieved highest overall F1-scores.

Bio_ClinicalBERT had high precision but low recall.

Model performance varied significantly across categories.

Abstract

Understanding how individuals with Parkinson's disease (PD) describe cognitive experiences in their daily lives can offer valuable insights into disease-related cognitive and emotional changes. However, extracting such information from unstructured patient narratives is challenging due to the subtle, overlapping nature of cognitive constructs. This study developed and evaluated natural language processing (NLP) models to automatically identify categories that reflect various cognitive processes from de-identified first-person narratives. Three model families, a Bio_ClinicalBERT-based span categorization model for nested entity recognition, a fine-tuned Meta-Llama-3-8B-Instruct model using QLoRA for instruction following, and GPT-4o mini evaluated under zero- and few-shot settings, were compared on their performance on extracting seven categories. Our findings indicated that model performance varied substantially across categories and model families. The fine-tuned Meta-Llama-3-8B-Instruct achieved the highest overall F1-scores (0.74 micro-average and 0.59 macro-average), particularly excelling in context-dependent categories such as thought and social interaction. Bio_ClinicalBERT exhibited high precision but low recall and performed comparable to Llama for some category types such as location and time but failed on other categories such as thought, emotion and social interaction. Compared to conventional information extraction tasks, this task presents a greater challenge due to the abstract and overlapping nature of narrative accounts of complex cognitive processes. Nonetheless, with continued refinement, these NLP systems hold promise for enabling low-burden, longitudinal monitoring of cognitive function and serving as a valuable complement to formal neuropsychological assessments in PD.