Towards Retrieval-Augmented Architectures for Image Captioning

TL;DR

This paper introduces retrieval-augmented image captioning models that leverage external visual memory to enhance caption quality, demonstrating significant improvements on COCO and nocaps datasets.

Contribution

It presents a novel retrieval-augmented architecture with external memory and a knowledge retriever, advancing image captioning methods beyond traditional deep learning models.

Findings

External memory improves caption quality significantly.

Larger retrieval corpus yields better results.

Models outperform baseline on COCO and nocaps datasets.

Abstract

The objective of image captioning models is to bridge the gap between the visual and linguistic modalities by generating natural language descriptions that accurately reflect the content of input images. In recent years, researchers have leveraged deep learning-based models and made advances in the extraction of visual features and the design of multimodal connections to tackle this task. This work presents a novel approach towards developing image captioning models that utilize an external kNN memory to improve the generation process. Specifically, we propose two model variants that incorporate a knowledge retriever component that is based on visual similarities, a differentiable encoder to represent input images, and a kNN-augmented language model to predict tokens based on contextual cues and text retrieved from the external memory. We experimentally validate our approach on COCO and nocaps datasets and demonstrate that incorporating an explicit external memory can significantly enhance the quality of captions, especially with a larger retrieval corpus. This work provides valuable insights into retrieval-augmented captioning models and opens up new avenues for improving image captioning at a larger scale.