UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning

TL;DR

UMIC is a novel unreferenced image captioning metric leveraging contrastive learning with Vision-and-Language BERT, outperforming existing metrics in correlation without needing reference captions.

Contribution

Introduces UMIC, a reference-free image captioning evaluation metric trained with contrastive learning, and provides a new human-annotated benchmark dataset.

Findings

UMIC outperforms previous metrics in correlation with human judgments.

UMIC does not require reference captions for evaluation.

A new human-annotated dataset for image captioning is released.

Abstract

Despite the success of various text generation metrics such as BERTScore, it is still difficult to evaluate the image captions without enough reference captions due to the diversity of the descriptions. In this paper, we introduce a new metric UMIC, an Unreferenced Metric for Image Captioning which does not require reference captions to evaluate image captions. Based on Vision-and-Language BERT, we train UMIC to discriminate negative captions via contrastive learning. Also, we observe critical problems of the previous benchmark dataset (i.e., human annotations) on image captioning metric, and introduce a new collection of human annotations on the generated captions. We validate UMIC on four datasets, including our new dataset, and show that UMIC has a higher correlation than all previous metrics that require multiple references. We release the benchmark dataset and pre-trained models to compute the UMIC.