Streaming Generation for Music Accompaniment

TL;DR

This paper introduces a real-time audio-to-audio music accompaniment model that balances latency, coherence, and throughput, addressing system delays and proposing advanced training objectives for live performance scenarios.

Contribution

It presents a novel model design considering system delays, explores the trade-offs between future visibility and chunk size, and highlights the need for anticipatory training objectives for coherent live accompaniment.

Findings

Increasing future visibility improves coherence but demands faster inference.

Larger output chunks increase throughput but reduce update frequency and quality.

Naive training methods are insufficient for real-time coherent accompaniment.

Abstract

Music generation models can produce high-fidelity coherent accompaniment given complete audio input, but are limited to editing and loop-based workflows. We study real-time audio-to-audio accompaniment: as a model hears an input audio stream (e.g., a singer singing), it has to also simultaneously generate in real-time a coherent accompanying stream (e.g., a guitar accompaniment). In this work, we propose a model design considering inevitable system delays in practical deployment with two design variables: future visibility $t_f$, the offset between the output playback time and the latest input time used for conditioning, and output chunk duration $k$, the number of frames emitted per call. We train Transformer decoders across a grid of $(t_f,k)$ and show two consistent trade-offs: increasing effective $t_f$ improves coherence by reducing the recency gap, but requires faster inference to stay within the latency budget; increasing $k$ improves throughput but results in degraded accompaniment due to a reduced update rate. Finally, we observe that naive maximum-likelihood streaming training is insufficient for coherent accompaniment where future context is not available, motivating advanced anticipatory and agentic objectives for live jamming.