Contrastive Learning for Image Captioning

TL;DR

This paper introduces a contrastive learning approach to improve the distinctiveness of image captions without sacrificing quality, demonstrating significant improvements across multiple datasets.

Contribution

The paper presents a novel contrastive learning method that enhances caption distinctiveness in image captioning models, applicable to various architectures.

Findings

Significant improvement in caption distinctiveness on benchmark datasets

Method is compatible with different model structures

Enhances caption quality while increasing uniqueness

Abstract

Image captioning, a popular topic in computer vision, has achieved substantial progress in recent years. However, the distinctiveness of natural descriptions is often overlooked in previous work. It is closely related to the quality of captions, as distinctive captions are more likely to describe images with their unique aspects. In this work, we propose a new learning method, Contrastive Learning (CL), for image captioning. Specifically, via two constraints formulated on top of a reference model, the proposed method can encourage distinctiveness, while maintaining the overall quality of the generated captions. We tested our method on two challenging datasets, where it improves the baseline model by significant margins. We also showed in our studies that the proposed method is generic and can be used for models with various structures.