The Solution for the CVPR2023 NICE Image Captioning Challenge

TL;DR

This paper presents a comprehensive solution for the CVPR2023 NICE Image Captioning Challenge, utilizing large-scale data, contrastive learning, and retrieval-augmented strategies to generate high-quality, semantically-rich captions for diverse images.

Contribution

The paper introduces a novel combination of external large-scale data, contrastive learning, similarity-bucket strategy, and retrieval-augmented templates for improved zero-shot image captioning.

Findings

Achieved top ranking with 105.17 Cider-Score in validation.

Utilized Laion-5B dataset for diverse training data.

Incorporated retrieval-augmented templates for richer captions.

Abstract

In this paper, we present our solution to the New frontiers for Zero-shot Image Captioning Challenge. Different from the traditional image captioning datasets, this challenge includes a larger new variety of visual concepts from many domains (such as COVID-19) as well as various image types (photographs, illustrations, graphics). For the data level, we collect external training data from Laion-5B, a large-scale CLIP-filtered image-text dataset. For the model level, we use OFA, a large-scale visual-language pre-training model based on handcrafted templates, to perform the image captioning task. In addition, we introduce contrastive learning to align image-text pairs to learn new visual concepts in the pre-training stage. Then, we propose a similarity-bucket strategy and incorporate this strategy into the template to force the model to generate higher quality and more matching captions. Finally, by retrieval-augmented strategy, we construct a content-rich template, containing the most relevant top-k captions from other image-text pairs, to guide the model in generating semantic-rich captions. Our method ranks first on the leaderboard, achieving 105.17 and 325.72 Cider-Score in the validation and test phase, respectively.