Presto! Distilling Steps and Layers for Accelerating Music Generation

TL;DR

Presto! introduces a dual distillation approach that significantly accelerates diffusion-based text-to-music generation, achieving state-of-the-art speed and quality improvements through novel step and layer distillation methods.

Contribution

The paper presents the first GAN-based distillation for TTM and combines step and layer distillation techniques to greatly enhance inference speed and output quality.

Findings

Achieves 10-18x faster music generation

Produces high-quality, diverse music outputs

Outperforms existing state-of-the-art methods in speed and quality

Abstract

Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) -- the fastest high-quality TTM to our knowledge. Sound examples can be found at https://presto-music.github.io/web/.