Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates

TL;DR

This paper introduces SPIDEr-max, a new evaluation metric for automatic audio captioning that considers multiple candidate captions and shows it can better reflect system performance by aligning closer to human scores.

Contribution

The paper proposes SPIDEr-max, a novel metric that evaluates multiple caption candidates by their maximum score, improving assessment of AAC systems.

Findings

SPIDEr-max correlates well with human judgment.

Using multiple candidates provides a more comprehensive evaluation.

System performance with SPIDEr-max approaches human scores.

Abstract

Automatic Audio Captioning (AAC) is the task that aims to describe an audio signal using natural language. AAC systems take as input an audio signal and output a free-form text sentence, called a caption. Evaluating such systems is not trivial, since there are many ways to express the same idea. For this reason, several complementary metrics, such as BLEU, CIDEr, SPICE and SPIDEr, are used to compare a single automatic caption to one or several captions of reference, produced by a human annotator. Nevertheless, an automatic system can produce several caption candidates, either using some randomness in the sentence generation process, or by considering the various competing hypothesized captions during decoding with beam-search, for instance. If we consider an end-user of an AAC system, presenting several captions instead of a single one seems relevant to provide some diversity, similarly to information retrieval systems. In this work, we explore the possibility to consider several predicted captions in the evaluation process instead of one. For this purpose, we propose SPIDEr-max, a metric that takes the maximum SPIDEr value among the scores of several caption candidates. To advocate for our metric, we report experiments on Clotho v2.1 and AudioCaps, with a transformed-based system. On AudioCaps for example, this system reached a SPIDEr-max value (with 5 candidates) close to the SPIDEr human score of reference.