Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis

TL;DR

This paper provides a comprehensive survey and taxonomy of over 70 image captioning evaluation metrics, revealing reliance on few metrics and proposing an ensemble method to better align with human judgments.

Contribution

It offers the first detailed taxonomy of image captioning metrics and introduces EnsembEval, a linear regression ensemble that improves correlation with human ratings.

Findings

Most studies rely on only five metrics

Existing metrics are weakly correlated with human judgments

EnsembEval improves correlation across multiple datasets

Abstract

The task of image captioning has recently been gaining popularity, and with it the complex task of evaluating the quality of image captioning models. In this work, we present the first survey and taxonomy of over 70 different image captioning metrics and their usage in hundreds of papers, specifically designed to help users select the most suitable metric for their needs. We find that despite the diversity of proposed metrics, the vast majority of studies rely on only five popular metrics, which we show to be weakly correlated with human ratings. We hypothesize that combining a diverse set of metrics can enhance correlation with human ratings. As an initial step, we demonstrate that a linear regression-based ensemble method, which we call EnsembEval, trained on one human ratings dataset, achieves improved correlation across five additional datasets, showing there is a lot of room for improvement by leveraging a diverse set of metrics.