Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation

TL;DR

JASCO is a novel text-to-music generation model that combines symbolic and audio conditions for fine-grained, controllable music synthesis, leveraging flow matching and information bottleneck techniques.

Contribution

The paper introduces JASCO, a new model that integrates symbolic and audio-based controls for temporally precise music generation using a flow matching framework.

Findings

JASCO achieves comparable quality to baseline models.

JASCO provides significantly better control over generated music.

Human evaluations favor JASCO's controllability.

Abstract

We present JASCO, a temporally controlled text-to-music generation model utilizing both symbolic and audio-based conditions. JASCO can generate high-quality music samples conditioned on global text descriptions along with fine-grained local controls. JASCO is based on the Flow Matching modeling paradigm together with a novel conditioning method. This allows music generation controlled both locally (e.g., chords) and globally (text description). Specifically, we apply information bottleneck layers in conjunction with temporal blurring to extract relevant information with respect to specific controls. This allows the incorporation of both symbolic and audio-based conditions in the same text-to-music model. We experiment with various symbolic control signals (e.g., chords, melody), as well as with audio representations (e.g., separated drum tracks, full-mix). We evaluate JASCO considering both generation quality and condition adherence, using both objective metrics and human studies. Results suggest that JASCO is comparable to the evaluated baselines considering generation quality while allowing significantly better and more versatile controls over the generated music. Samples are available on our demo page https://pages.cs.huji.ac.il/adiyoss-lab/JASCO.