SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusion

TL;DR

SAMUeL introduces a lightweight, efficient vocal-conditioned music generation model using soft alignment attention and latent diffusion, enabling real-time, high-quality music synthesis on consumer hardware.

Contribution

The paper proposes a novel soft alignment attention mechanism within a latent diffusion framework, significantly reducing model size and inference time while maintaining high-quality music generation.

Findings

220x fewer parameters than state-of-the-art systems

52x faster inference speed

Outperforms OpenAI Jukebox in quality and coherence

Abstract

We present a lightweight latent diffusion model for vocal-conditioned musical accompaniment generation that addresses critical limitations in existing music AI systems. Our approach introduces a novel soft alignment attention mechanism that adaptively combines local and global temporal dependencies based on diffusion timesteps, enabling efficient capture of multi-scale musical structure. Operating in the compressed latent space of a pre-trained variational autoencoder, the model achieves a 220 times parameter reduction compared to state-of-the-art systems while delivering 52 times faster inference. Experimental evaluation demonstrates competitive performance with only 15M parameters, outperforming OpenAI Jukebox in production quality and content unity while maintaining reasonable musical coherence. The ultra-lightweight architecture enables real-time deployment on consumer hardware, making AI-assisted music creation accessible for interactive applications and resource-constrained environments.