Toward a Sparse and Interpretable Audio Codec

TL;DR

This paper proposes a novel audio encoding method that represents sound as sparse events with physical context, aiming for interpretability and efficiency over traditional block-based codecs.

Contribution

It introduces a proof-of-concept encoder that models audio as sparse events using physics-inspired assumptions, enhancing interpretability and sparsity.

Findings

Demonstrates a sparse, event-based audio representation

Uses physics-based assumptions to model audio features

Encourages interpretability and efficiency in audio coding

Abstract

Most widely-used modern audio codecs, such as Ogg Vorbis and MP3, as well as more recent "neural" codecs like Meta's Encodec or the Descript Audio Codec are based on block-coding; audio is divided into overlapping, fixed-size "frames" which are then compressed. While they often yield excellent reproductions and can be used for downstream tasks such as text-to-audio, they do not produce an intuitive, directly-interpretable representation. In this work, we introduce a proof-of-concept audio encoder that represents audio as a sparse set of events and their times-of-occurrence. Rudimentary physics-based assumptions are used to model attack and the physical resonance of both the instrument being played and the room in which a performance occurs, hopefully encouraging a sparse, parsimonious, and easy-to-interpret representation.