On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches

TL;DR

This paper reveals that the UA-Speech and TORGO databases may not reliably validate dysarthric speech classification methods because they contain environmental artifacts, leading to potential misinterpretation of results.

Contribution

The study demonstrates that current databases may bias dysarthria classification results due to environmental factors, emphasizing the need for careful data quality considerations.

Findings

State-of-the-art methods perform well on non-speech segments.

Recording environment influences classification performance.

Databases may contain environmental artifacts affecting results.

Abstract

Although the UA-Speech and TORGO databases of control and dysarthric speech are invaluable resources made available to the research community with the objective of developing robust automatic speech recognition systems, they have also been used to validate a considerable number of automatic dysarthric speech classification approaches. Such approaches typically rely on the underlying assumption that recordings from control and dysarthric speakers are collected in the same noiseless environment using the same recording setup. In this paper, we show that this assumption is violated for the UA-Speech and TORGO databases. Using voice activity detection to extract speech and non-speech segments, we show that the majority of state-of-the-art dysarthria classification approaches achieve the same or a considerably better performance when using the non-speech segments of these databases than when using the speech segments. These results demonstrate that such approaches trained and validated on the UA-Speech and TORGO databases are potentially learning characteristics of the recording environment or setup rather than dysarthric speech characteristics. We hope that these results raise awareness in the research community about the importance of the quality of recordings when developing and evaluating automatic dysarthria classification approaches.