A Generative-First Neural Audio Autoencoder

TL;DR

This paper presents a generative-first neural audio autoencoder that significantly improves encoding speed, reduces latent rates, and unifies representations across formats, enabling more efficient and versatile audio generative modeling.

Contribution

It introduces a novel architecture that increases temporal downsampling and supports multiple formats in one model, balancing compression, quality, and speed.

Findings

10x faster encoding compared to previous methods

1.6x lower latent rates while maintaining quality

Unified model for various audio formats

Abstract

Neural autoencoders underpin generative models. Practical, large-scale use of neural autoencoders for generative modeling necessitates fast encoding, low latent rates, and a single model across representations. Existing approaches are reconstruction-first: they incur high latent rates, slow encoding, and separate architectures for discrete vs. continuous latents and for different audio channel formats, hindering workflows from preprocessing to inference conditioning. We introduce a generative-first architecture for audio autoencoding that increases temporal downsampling from 2048x to 3360x and supports continuous and discrete representations and common audio channel formats in one model. By balancing compression, quality, and speed, it delivers 10x faster encoding, 1.6x lower rates, and eliminates channel-format-specific variants while maintaining competitive reconstruction quality. This enables applications previously constrained by processing costs: a 60-second mono signal compresses to 788 tokens, making generative modeling more tractable.