Pitfalls and Limits in Automatic Dementia Assessment

TL;DR

This paper critically examines speech-based dementia assessment methods, revealing biases and artifacts that affect accuracy across different impairment levels, emphasizing the need for detailed error analysis beyond numerical performance.

Contribution

It provides an in-depth analysis of an automated dementia assessment, highlighting biases and pitfalls that influence the reliability of speech-based evaluation tools.

Findings

High correlation with human scores for severely impaired individuals

Speech production decreases with cognitive decline, affecting test scoring

Fallback handling introduces biases favoring certain groups

Abstract

Current work on speech-based dementia assessment focuses on either feature extraction to predict assessment scales, or on the automation of existing test procedures. Most research uses public data unquestioningly and rarely performs a detailed error analysis, focusing primarily on numerical performance. We perform an in-depth analysis of an automated standardized dementia assessment, the Syndrom-Kurz-Test. We find that while there is a high overall correlation with human annotators, due to certain artifacts, we observe high correlations for the severely impaired individuals, which is less true for the healthy or mildly impaired ones. Speech production decreases with cognitive decline, leading to overoptimistic correlations when test scoring relies on word naming. Depending on the test design, fallback handling introduces further biases that favor certain groups. These pitfalls remain independent of group distributions in datasets and require differentiated analysis of target groups.