Reading Between the Lines: Combining Pause Dynamics and Semantic Coherence for Automated Assessment of Thought Disorder

TL;DR

This study develops a scalable multimodal framework combining pause dynamics and semantic coherence to improve automated assessment of thought disorder severity in speech, showing consistent performance gains across diverse datasets.

Contribution

It introduces an integrated approach that combines pause features with semantic measures, demonstrating enhanced predictive accuracy for thought disorder severity over semantic measures alone.

Findings

Pause features alone predict FTD severity robustly.

Combining pause and semantic features improves prediction accuracy.

Late fusion yields the most consistent performance gains.

Abstract

Formal thought disorder (FTD), a hallmark of schizophrenia spectrum disorders, manifests as incoherent speech and poses challenges for clinical assessment. Traditional clinical rating scales, though validated, are resource-intensive and lack scalability. Automated speech recognition (ASR) allows for objective quantification of linguistic and temporal features of speech, offering scalable alternatives. Furthermore, ASR-derived utterance timestamps provide access to pause dynamics, which are thought to reflect the cognitive processes underlying speech production. Yet, their added value beyond semantic measures remains insufficiently explored. In this study, we evaluated a scalable multimodal framework that integrates pause features with semantic coherence metrics across three datasets: naturalistic self-recorded diaries (AVH), structured picture descriptions (TOPSY), and dream narratives (PsyCL). Pause-related features were evaluated alongside established coherence measures using support vector regression to predict clinical FTD scores. Models using pause features alone robustly predict manually rated FTD severity consistently across datasets. Integrating pause features with semantic coherence metrics enhanced predictive performance compared to coherence-only models, with late fusion yielding the most robust and consistent gains in all three datasets. On average across datasets, Spearman correlation increased from \r{ho} = 0.413 for semantic-only models to \r{ho} = 0.455 with late fusion. The performance gains from semantic and pause features integration held consistently across all contexts, though the nature of the most informative pause patterns was dataset-dependent. These findings suggest that both pause dynamics and semantic coherence reflect complementary aspects of thought disorganization.