RELATE: Subjective evaluation dataset for automatic evaluation of relevance between text and audio

TL;DR

This paper introduces RELATE, a new dataset for subjective relevance evaluation in text-to-audio tasks, and benchmarks a model that predicts subjective scores more accurately than existing methods.

Contribution

The paper presents RELATE, an open-source dataset for subjective relevance evaluation, and demonstrates a model that better predicts human relevance scores than traditional approaches.

Findings

The proposed model outperforms CLAPScore in relevance prediction.

RELATE dataset enables more accurate subjective relevance assessment.

Model performance extends across various sound categories.

Abstract

In text-to-audio (TTA) research, the relevance between input text and output audio is an important evaluation aspect. Traditionally, it has been evaluated from both subjective and objective perspectives. However, subjective evaluation is costly in terms of money and time, and objective evaluation is unclear regarding the correlation to subjective evaluation scores. In this study, we construct RELATE, an open-sourced dataset that subjectively evaluates the relevance. Also, we benchmark a model for automatically predicting the subjective evaluation score from synthesized audio. Our model outperforms a conventional CLAPScore model, and that trend extends to many sound categories.