WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction

TL;DR

WhisQ is a novel multimodal model that predicts music quality and text alignment by combining sequence co-attention and optimal transport, significantly improving evaluation accuracy in text-to-music systems.

Contribution

The paper introduces WhisQ, a new cross-modal architecture utilizing sequence co-attention and optimal transport regularization for improved MOS prediction in text-to-music tasks.

Findings

7% improvement in Spearman correlation for overall music quality

14% improvement in text alignment accuracy

Optimal transport regularization yields 10% SRCC gain

Abstract

Mean Opinion Score (MOS) prediction for text to music systems requires evaluating both overall musical quality and text prompt alignment. This paper introduces WhisQ, a multimodal architecture that addresses this dual-assessment challenge through sequence level co-attention and optimal transport regularization. WhisQ employs the Whisper Base pretrained model for temporal audio encoding and Qwen 3, a 0.6B Small Language Model (SLM), for text encoding, with both maintaining sequence structure for fine grained cross-modal modeling. The architecture features specialized prediction pathways: OMQ is predicted from pooled audio embeddings, while TA leverages bidirectional sequence co-attention between audio and text. Sinkhorn optimal transport loss further enforce semantic alignment in the shared embedding space. On the MusicEval Track-1 dataset, WhisQ achieves substantial improvements over the baseline: 7% improvement in Spearman correlation for OMQ and 14% for TA. Ablation studies reveal that optimal transport regularization provides the largest performance gain (10% SRCC improvement), demonstrating the importance of explicit cross-modal alignment for text-to-music evaluation.