SUNAC: Source-aware Unified Neural Audio Codec

TL;DR

SUNAC is a neural audio codec that encodes individual sources from mixtures based on source prompts, enabling efficient source-specific processing with competitive quality and lower computational cost.

Contribution

It introduces a source-aware neural audio codec that encodes sources directly from mixtures conditioned on prompts, improving flexibility and efficiency.

Findings

Achieves competitive resynthesis quality

Enables user-driven source selection

Reduces computational cost compared to cascaded methods

Abstract

Neural audio codecs (NACs) provide compact representations that can be leveraged in many downstream applications, in particular large language models. Yet most NACs encode mixtures of multiple sources in an entangled manner, which may impede efficient downstream processing in applications that need access to only a subset of the sources (e.g., analysis of a particular type of sound, transcription of a given speaker, etc). To address this, we propose a source-aware codec that encodes individual sources directly from mixtures, conditioned on source type prompts. This enables user-driven selection of which source(s) to encode, including separately encoding multiple sources of the same type (e.g., multiple speech signals). Experiments show that our model achieves competitive resynthesis and separation quality relative to a cascade of source separation followed by a conventional NAC, with lower computational cost.