Human-CLAP: Human-perception-based contrastive language-audio pretraining

TL;DR

This paper introduces Human-CLAP, a contrastive language-audio model trained on human subjective scores, significantly improving the correlation between model scores and human perception in audio-text relevance evaluation.

Contribution

The paper proposes Human-CLAP, a novel contrastive model trained with human subjective scores, enhancing the alignment between model metrics and human perception.

Findings

Human-CLAP increases SRCC with subjective scores by over 0.25.

CLAPScore has low correlation with human subjective evaluation.

Human-CLAP improves the reliability of audio-text relevance assessment.

Abstract

Contrastive language-audio pretraining (CLAP) is widely used for audio generation and recognition tasks. For example, CLAPScore, which utilizes the similarity of CLAP embeddings, has been a major metric for the evaluation of the relevance between audio and text in text-to-audio. However, the relationship between CLAPScore and human subjective evaluation scores is still unclarified. We show that CLAPScore has a low correlation with human subjective evaluation scores. Additionally, we propose a human-perception-based CLAP called Human-CLAP by training a contrastive language-audio model using the subjective evaluation score. In our experiments, the results indicate that our Human-CLAP improved the Spearman's rank correlation coefficient (SRCC) between the CLAPScore and the subjective evaluation scores by more than 0.25 compared with the conventional CLAP.