Layer-Aware Early Fusion of Acoustic and Linguistic Embeddings for Cognitive Status Classification

TL;DR

This study explores how layer-aware early fusion of acoustic and linguistic embeddings from speech and transcription improves cognitive status classification, revealing optimal layer combinations and the importance of modality-specific fusion strategies.

Contribution

It introduces a layer-aware early fusion approach that enhances multimodal speech analysis for cognitive decline detection, emphasizing the significance of internal layer selection.

Findings

Mid-layer embeddings (layers 8-10) yield peak performance.

Acoustic-only models outperform text-only models.

Early fusion enhances acoustic modality discrimination.

Abstract

Speech contains both acoustic and linguistic patterns that reflect cognitive decline, and therefore models describing only one domain cannot fully capture such complexity. This study investigates how early fusion (EF) of speech and its corresponding transcription text embeddings, with attention to encoder layer depth, can improve cognitive status classification. Using a DementiaBank-derived collection of recordings (1,629 speakers; cognitively normal controls$\unicode{x2013}$CN, Mild Cognitive Impairment$\unicode{x2013}$MCI, and Alzheimer's Disease and Related Dementias$\unicode{x2013}$ADRD), we extracted frame-aligned embeddings from different internal layers of wav2vec 2.0 or Whisper combined with DistilBERT or RoBERTa. Unimodal, EF and late fusion (LF) models were trained with a transformer classifier, optimized, and then evaluated across 10 seeds. Performance consistently peaked in mid encoder layers ($\sim$8$\unicode{x2013}$10), with the single best F1 at Whisper + RoBERTa layer 9 and the best log loss at Whisper + DistilBERT layer 10. Acoustic-only models consistently outperformed text-only variants. EF boosts discrimination for genuinely acoustic embeddings, whereas LF improves probability calibration. Layer choice critically shapes clinical multimodal synergy.