It's Raw! Audio Generation with State-Space Models

TL;DR

SaShiMi introduces a novel multi-scale waveform modeling architecture based on S4, achieving state-of-the-art results in unconditional audio generation, outperforming prior models like WaveNet in quality and efficiency.

Contribution

The paper presents SaShiMi, a new architecture for raw audio modeling that improves stability, performance, and efficiency over existing models by leveraging the S4 model and novel parameterization techniques.

Findings

SaShiMi achieves 2x better mean opinion scores than WaveNet for speech.

SaShiMi outperforms WaveNet in density estimation and speed with fewer parameters.

SaShiMi improves both autoregressive and non-autoregressive audio generation tasks.

Abstract

Developing architectures suitable for modeling raw audio is a challenging problem due to the high sampling rates of audio waveforms. Standard sequence modeling approaches like RNNs and CNNs have previously been tailored to fit the demands of audio, but the resultant architectures make undesirable computational tradeoffs and struggle to model waveforms effectively. We propose SaShiMi, a new multi-scale architecture for waveform modeling built around the recently introduced S4 model for long sequence modeling. We identify that S4 can be unstable during autoregressive generation, and provide a simple improvement to its parameterization by drawing connections to Hurwitz matrices. SaShiMi yields state-of-the-art performance for unconditional waveform generation in the autoregressive setting. Additionally, SaShiMi improves non-autoregressive generation performance when used as the backbone architecture for a diffusion model. Compared to prior architectures in the autoregressive generation setting, SaShiMi generates piano and speech waveforms which humans find more musical and coherent respectively, e.g. 2x better mean opinion scores than WaveNet on an unconditional speech generation task. On a music generation task, SaShiMi outperforms WaveNet on density estimation and speed at both training and inference even when using 3x fewer parameters. Code can be found at https://github.com/HazyResearch/state-spaces and samples at https://hazyresearch.stanford.edu/sashimi-examples.