Clinical Annotations for Automatic Stuttering Severity Assessment

TL;DR

This paper enhances the FluencyBank dataset with expert-annotated, multi-modal clinical data for automatic stuttering severity assessment, emphasizing the importance of clinical expertise for accurate model training and evaluation.

Contribution

It introduces a new clinical annotation scheme for stuttering, incorporating audiovisual features and expert consensus to improve assessment models.

Findings

Annotations reflect real-world clinical standards

Expert consensus provides highly reliable test data

Complexity of stuttering assessment requires clinical expertise

Abstract

Stuttering is a complex disorder that requires specialized expertise for effective assessment and treatment. This paper presents an effort to enhance the FluencyBank dataset with a new stuttering annotation scheme based on established clinical standards. To achieve high-quality annotations, we hired expert clinicians to label the data, ensuring that the resulting annotations mirror real-world clinical expertise. The annotations are multi-modal, incorporating audiovisual features for the detection and classification of stuttering moments, secondary behaviors, and tension scores. In addition to individual annotations, we additionally provide a test set with highly reliable annotations based on expert consensus for assessing individual annotators and machine learning models. Our experiments and analysis illustrate the complexity of this task that necessitates extensive clinical expertise for valid training and evaluation of stuttering assessment models.