HCLAS-X: Hierarchical and Cascaded Lyrics Alignment System Using Multimodal Cross-Correlation

TL;DR

HCLAS-X is a hierarchical, cascaded system for aligning lyrics with vocals in songs, leveraging cross-correlation of latent representations to improve accuracy across long sequences.

Contribution

The paper introduces a novel hierarchical and cascaded model that uses cross-correlation for lyrics-vocal alignment, enabling effective processing of long sequences and outperforming previous models.

Findings

Significant reduction in mean average error.

Robustness demonstrated in real-world music streaming applications.

Effective handling of long sequence alignment tasks.

Abstract

In this work, we address the challenge of lyrics alignment, which involves aligning the lyrics and vocal components of songs. This problem requires the alignment of two distinct modalities, namely text and audio. To overcome this challenge, we propose a model that is trained in a supervised manner, utilizing the cross-correlation matrix of latent representations between vocals and lyrics. Our system is designed in a hierarchical and cascaded manner. It predicts synced time first on a sentence-level and subsequently on a word-level. This design enables the system to process long sequences, as the cross-correlation uses quadratic memory with respect to sequence length. In our experiments, we demonstrate that our proposed system achieves a significant improvement in mean average error, showcasing its robustness in comparison to the previous state-of-the-art model. Additionally, we conduct a qualitative analysis of the system after successfully deploying it in several music streaming services.