WavFlow: Audio Generation in Waveform Space

TL;DR

WavFlow introduces a novel approach for high-fidelity audio generation directly in waveform space, bypassing traditional latent compression, and achieves competitive results on standard benchmarks.

Contribution

This work presents WavFlow, a direct waveform generation framework that simplifies audio synthesis and matches state-of-the-art performance without relying on intermediate representations.

Findings

WavFlow achieves competitive scores on VGGSound and AudioCaps benchmarks.

The model can learn fine-grained acoustic patterns from scratch using large-scale video-text-audio data.

Direct waveform modeling can match or surpass latent-space methods in audio quality.

Abstract

Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.