Improving Reference-based Distinctive Image Captioning with Contrastive Rewards

TL;DR

This paper introduces a new contrastive learning module and evaluation metric for reference-based distinctive image captioning, improving the ability to generate unique, accurate captions that highlight image-specific details.

Contribution

It proposes a contrastive learning module integrated into Transformer-based models and introduces new benchmarks and metrics for more effective evaluation of distinctive captioning.

Findings

TransDIC++ outperforms state-of-the-art models on new benchmarks

The contrastive module enhances the perception of unique image attributes

DisCIDEr metric effectively evaluates caption accuracy and distinctiveness

Abstract

Distinctive Image Captioning (DIC) -- generating distinctive captions that describe the unique details of a target image -- has received considerable attention over the last few years. A recent DIC method proposes to generate distinctive captions by comparing the target image with a set of semantic-similar reference images, i.e., reference-based DIC (Ref-DIC). It aims to force the generated captions to distinguish between the target image and the reference image. To ensure Ref-DIC models really perceive the unique objects (or attributes) in target images, we propose two new Ref-DIC benchmarks and develop a Transformer-based Ref-DIC baseline TransDIC. The model only extracts visual features from the target image, but also encodes the differences between objects in the target and reference images. Taking one step further, we propose a stronger TransDIC++, which consists of an extra contrastive learning module to make full use of the reference images. This new module is model-agnostic, which can be easily incorporated into various Ref-DIC architectures. Finally, for more trustworthy benchmarking, we propose a new evaluation metric named DisCIDEr for Ref-DIC, which evaluates both the accuracy and distinctiveness of the generated captions. Experimental results demonstrate that our TransDIC++ can generate distinctive captions. Besides, it outperforms several state-of-the-art models on the two new benchmarks over different metrics.