Modeling Image-Caption Rating from Comparative Judgments

TL;DR

This paper introduces a machine learning framework that models comparative judgments for image-caption rating, resulting in efficient, consistent, and high-performing ranking of image-caption pairs, reducing annotation costs.

Contribution

It proposes a novel framework that trains models on comparative judgments instead of direct ratings, improving efficiency and consistency in image-caption evaluation.

Findings

The regression model achieved Kendall's τc=0.812 on VICR dataset.

The comparative learning model achieved Kendall's τc=0.804, comparable to the regression model.

Comparative judgments are faster and more consistent than direct ratings in human studies.

Abstract

Image caption rating is becoming increasingly important because computer-generated captions are used extensively for descriptive annotation. However, rating the accuracy of captions in describing images is time-consuming and subjective in nature. In contrast, it is often easier for people to compare (between two pairs) which image-caption pair better matches each other. In this study, we propose a machine learning framework that models such comparative judgments instead of direct ratings. The model can then be applied to rank unseen image-caption pairs in the same way as a regression model trained on direct ratings. Inspired by a state-of-the-art regression approach, we extracted visual and text features using a pre-trained ViLBERT model and tweaked the learning parameters of the baseline model to improve the model performance. This new regression model (with Kendall's $\tau_c=0.812$) outperformed the baseline model (with Kendall's $\tau_c=0.758$) on the VICR dataset. The same model structure was applied to the comparative learning framework. Trained on comparative judgments (image-caption pair A better matches each other than image-caption pair B), the comparative learning model achieved a performance similar (with Kendall's $\tau_c=0.804$) to that of the regression model. In addition, a small-scale human subject study was conducted to compare the cost and quality of direct ratings, pairwise comparisons, and same-image comparisons. The results showed that comparative judgments yielded faster results and greater agreement among human annotators than direct ratings. These results suggest that collecting comparative judgments instead of direct ratings as training data labels is promising for lower annotation costs and greater consistency. The model trained on such comparative judgments can perform as well as the model trained on direct ratings.