On Distinctive Image Captioning via Comparing and Reweighting

TL;DR

This paper introduces a new metric and training strategy to enhance the distinctiveness of image captions by comparing them with similar images, leading to more unique and informative descriptions without sacrificing accuracy.

Contribution

It proposes CIDErBtw, a distinctiveness metric, and a reweighting training method that emphasizes rare words and uses negative sampling to generate more distinctive image captions.

Findings

Significantly improves caption distinctiveness as measured by CIDErBtw and retrieval metrics.

Enhances caption accuracy while increasing uniqueness.

Validated through extensive experiments and user studies.

Abstract

Recent image captioning models are achieving impressive results based on popular metrics, i.e., BLEU, CIDEr, and SPICE. However, focusing on the most popular metrics that only consider the overlap between the generated captions and human annotation could result in using common words and phrases, which lacks distinctiveness, i.e., many similar images have the same caption. In this paper, we aim to improve the distinctiveness of image captions via comparing and reweighting with a set of similar images. First, we propose a distinctiveness metric -- between-set CIDEr (CIDErBtw) to evaluate the distinctiveness of a caption with respect to those of similar images. Our metric reveals that the human annotations of each image in the MSCOCO dataset are not equivalent based on distinctiveness; however, previous works normally treat the human annotations equally during training, which could be a reason for generating less distinctive captions. In contrast, we reweight each ground-truth caption according to its distinctiveness during training. We further integrate a long-tailed weight strategy to highlight the rare words that contain more information, and captions from the similar image set are sampled as negative examples to encourage the generated sentence to be unique. Finally, extensive experiments are conducted, showing that our proposed approach significantly improves both distinctiveness (as measured by CIDErBtw and retrieval metrics) and accuracy (e.g., as measured by CIDEr) for a wide variety of image captioning baselines. These results are further confirmed through a user study.