Clever Hans Effect Found in Automatic Detection of Alzheimer's Disease through Speech

TL;DR

This paper reveals a bias in the Pitt corpus for Alzheimer's detection via speech, where silent segments alone can nearly perfectly identify AD, indicating a Clever Hans effect that questions model reliability.

Contribution

The study uncovers a dataset bias causing misleading high accuracy in AD detection models, emphasizing the need for careful dataset analysis and validation.

Findings

Silent segments can nearly perfectly predict AD in Pitt corpus.

Bias in datasets can lead to misleading model performance.

Clever Hans effect observed in speech-based AD detection.

Abstract

We uncover an underlying bias present in the audio recordings produced from the picture description task of the Pitt corpus, the largest publicly accessible database for Alzheimer's Disease (AD) detection research. Even by solely utilizing the silent segments of these audio recordings, we achieve nearly 100% accuracy in AD detection. However, employing the same methods to other datasets and preprocessed Pitt recordings results in typical levels (approximately 80%) of AD detection accuracy. These results demonstrate a Clever Hans effect in AD detection on the Pitt corpus. Our findings emphasize the crucial importance of maintaining vigilance regarding inherent biases in datasets utilized for training deep learning models, and highlight the necessity for a better understanding of the models' performance.