CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation

TL;DR

CAF-Score is a novel reference-free evaluation metric for audio captioning that combines CLAP embeddings with LALM reasoning to better detect syntactic errors and align with human judgments.

Contribution

It introduces CAF-Score, a new calibration method that enhances CLAP's semantic alignment with LALMs for more accurate audio captioning evaluation.

Findings

Achieves highest correlation with human judgments on BRACE benchmark.

Outperforms reference-based metrics in challenging scenarios.

Effectively detects syntactic inconsistencies and hallucinations.

Abstract

While Large Audio-Language Models (LALMs) have advanced audio captioning, robust evaluation remains difficult. Reference-based metrics are expensive and often fail to assess acoustic fidelity, while Contrastive Language-Audio Pretraining (CLAP)-based approaches frequently overlook syntactic errors and fine-grained details. We propose CAF-Score, a reference-free metric that calibrates CLAP's coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs. By combining contrastive audio-text embeddings with LALM reasoning, CAF-Score effectively detects syntactic inconsistencies and subtle hallucinations. Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments, even outperforming reference-based baselines in challenging scenarios. These results highlight the efficacy of CAF-Score for reference-free audio captioning evaluation. Code and results are available at https://github.com/inseong00/CAF-Score.