WavShape: Information-Theoretic Speech Representation Learning for Fair and Privacy-Aware Audio Processing

TL;DR

WavShape introduces an information-theoretic framework for learning speech representations that enhance fairness and privacy by reducing sensitive attribute information while maintaining task-relevant content, advancing fair and privacy-aware audio processing.

Contribution

It proposes a novel MI-based speech embedding method that systematically filters sensitive attributes, improving fairness and privacy without sacrificing task performance.

Findings

Reduces mutual information with sensitive attributes by up to 81%.

Retains 97% of task-relevant information.

Demonstrates effectiveness across three datasets.

Abstract

Speech embeddings often retain sensitive attributes such as speaker identity, accent, or demographic information, posing risks in biased model training and privacy leakage. We propose WavShape, an information-theoretic speech representation learning framework that optimizes embeddings for fairness and privacy while preserving task-relevant information. We leverage mutual information (MI) estimation using the Donsker-Varadhan formulation to guide an MI-based encoder that systematically filters sensitive attributes while maintaining speech content essential for downstream tasks. Experimental results on three known datasets show that WavShape reduces MI between embeddings and sensitive attributes by up to 81% while retaining 97% of task-relevant information. By integrating information theory with self-supervised speech models, this work advances the development of fair, privacy-aware, and resource-efficient speech systems.