Are metrics measuring what they should? An evaluation of image captioning task metrics

TL;DR

This paper evaluates the effectiveness of various image captioning metrics, including n-gram based and embedding-based methods, to determine how well they measure caption quality and semantic content.

Contribution

It provides a comprehensive comparison of current image captioning metrics using artificial and real caption data on the MS COCO dataset, highlighting their strengths and limitations.

Findings

Classical n-gram metrics are insufficient for capturing semantic content.

Embedding-based metrics like BERTScore and CLIPScore offer different insights.

Current metrics may not fully align with human judgment of caption quality.

Abstract

Image Captioning is a current research task to describe the image content using the objects and their relationships in the scene. To tackle this task, two important research areas converge, artificial vision, and natural language processing. In Image Captioning, as in any computational intelligence task, the performance metrics are crucial for knowing how well (or bad) a method performs. In recent years, it has been observed that classical metrics based on n-grams are insufficient to capture the semantics and the critical meaning to describe the content in an image. Looking to measure how well or not the set of current and more recent metrics are doing, in this article, we present an evaluation of several kinds of Image Captioning metrics and a comparison between them using the well-known MS COCO dataset. The metrics were selected from the most used in prior works, they are those based on $n$-grams as BLEU, SacreBLEU, METEOR, ROGUE-L, CIDEr, SPICE, and those based on embeddings, such as BERTScore and CLIPScore. For this, we designed two scenarios; 1) a set of artificially build captions with several qualities, and 2) a comparison of some state-of-the-art Image Captioning methods. Interesting findings were found trying to answer the questions: Are the current metrics helping to produce high-quality captions? How do actual metrics compare to each other? What are the metrics really measuring?