LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning

TL;DR

LAPS-Diff is a diffusion-based singing voice synthesis framework that incorporates language-aware embeddings and style-guided learning to improve naturalness and expressiveness in Bollywood Hindi singing, especially in low-resource settings.

Contribution

It introduces a novel diffusion model with integrated language, style, and pitch features, tailored for Hindi SVS, and demonstrates significant quality improvements over existing models.

Findings

Enhanced naturalness and expressiveness in synthesized singing.

Significant quality improvements over state-of-the-art models.

Effective use of language and style embeddings in low-resource scenarios.

Abstract

The field of Singing Voice Synthesis (SVS) has seen significant advancements in recent years due to the rapid progress of diffusion-based approaches. However, capturing vocal style, genre-specific pitch inflections, and language-dependent characteristics remains challenging, particularly in low-resource scenarios. To address this, we propose LAPS-Diff, a diffusion model integrated with language-aware embeddings and a vocal-style guided learning mechanism, specifically designed for Bollywood Hindi singing style. We curate a Hindi SVS dataset and leverage pre-trained language models to extract word and phone-level embeddings for an enriched lyrics representation. Additionally, we incorporated a style encoder and a pitch extraction model to compute style and pitch losses, capturing features essential to the naturalness and expressiveness of the synthesized singing, particularly in terms of vocal style and pitch variations. Furthermore, we utilize MERT and IndicWav2Vec models to extract musical and contextual embeddings, serving as conditional priors to refine the acoustic feature generation process further. Based on objective and subjective evaluations, we demonstrate that LAPS-Diff significantly improves the quality of the generated samples compared to the considered state-of-the-art (SOTA) model for our constrained dataset that is typical of the low resource scenario.