S2Accompanist: A Semantic-Aware and Structure-Guided Diffusion Model for Music Accompaniment Generation

TL;DR

S2Accompanist is a diffusion model that enhances music accompaniment generation by incorporating semantic awareness and structural guidance, achieving state-of-the-art results with limited data and computational resources.

Contribution

The paper introduces a novel semantic-aware and structure-guided diffusion model with an automated data pipeline and a specialized fine-tuning strategy for improved music accompaniment.

Findings

Achieved state-of-the-art performance on the ATTM Grand Challenge benchmark.

Secured first place in the Efficiency Track with only 402M parameters.

Demonstrated competitive results compared to larger models.

Abstract

High-fidelity text-to-music generation typically relies on massive proprietary datasets and immense computational resources. Existing models often struggle to generate coherent pure musical accompaniments and lack precise, localized semantic control due to their reliance on coarse, track-level annotations. To address these limitations under constrained data and computing resources, we propose S2Accompanist, a Semantic-Aware and Structure-Guided Diffusion Model developed for the ICME2026 ATTM Grand Challenge. Specifically, we design an automated data pipeline comprising structural segmentation, Large Audio-Language Model driven segment-level captioning, and dual-metric quality grading to overcome the absence of localized metadata in raw datasets. Furthermore, we propose a semantic-aware Variational Autoencoder fine-tuning strategy that explicitly distills foundational LeadSheet structures into the acoustic latent space, effectively improving the overall audio fidelity. Extensive experiments demonstrate that S2Accompanist achieves state-of-the-art objective performance on the ATTM Grand Challenge benchmark across both the Efficiency and Performance Tracks. With only 402M parameters, our model remains competitive compared to larger-scale unconstrained models and secured first place in the Efficiency Track.