Disentangling Textual and Acoustic Features of Neural Speech Representations

TL;DR

This paper introduces a disentanglement framework based on the Information Bottleneck principle to separate textual and acoustic features in neural speech models, aiding privacy and interpretability in speech processing tasks.

Contribution

It proposes a novel disentanglement method that isolates content and acoustic features in neural speech representations, enabling better analysis and privacy control.

Findings

Effective separation of textual and acoustic features demonstrated

Quantified feature contributions across model layers

Identified salient speech frames for downstream tasks

Abstract

Neural speech models build deeply entangled internal representations, which capture a variety of features (e.g., fundamental frequency, loudness, syntactic category, or semantic content of a word) in a distributed encoding. This complexity makes it difficult to track the extent to which such representations rely on textual and acoustic information, or to suppress the encoding of acoustic features that may pose privacy risks (e.g., gender or speaker identity) in critical, real-world applications. In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates complex speech representations into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a given downstream task. We apply and evaluate our framework to emotion recognition and speaker identification downstream tasks, quantifying the contribution of textual and acoustic features at each model layer. Additionally, we explore the application of our disentanglement framework as an attribution method to identify the most salient speech frame representations from both the textual and acoustic perspectives.