Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning

TL;DR

This paper enhances image captioning by integrating cross-modal textual and visual context, utilizing a multi-modal pre-trained model and additional relational information to improve grounding and caption quality.

Contribution

It introduces a novel approach that incorporates auxiliary relational data and a multi-modal pre-trained model to improve image captioning beyond fixed object detector outputs.

Findings

Achieved +7.5% CIDEr score improvement.

Demonstrated better grounding of objects in captions.

Validated the importance of multi-modal pre-trained models.

Abstract

Significant progress has been made on visual captioning, largely relying on pre-trained features and later fixed object detectors that serve as rich inputs to auto-regressive models. A key limitation of such methods, however, is that the output of the model is conditioned only on the object detector's outputs. The assumption that such outputs can represent all necessary information is unrealistic, especially when the detector is transferred across datasets. In this work, we reason about the graphical model induced by this assumption, and propose to add an auxiliary input to represent missing information such as object relationships. We specifically propose to mine attributes and relationships from the Visual Genome dataset and condition the captioning model on them. Crucially, we propose (and show to be important) the use of a multi-modal pre-trained model (CLIP) to retrieve such contextual descriptions. Further, object detector models are frozen and do not have sufficient richness to allow the captioning model to properly ground them. As a result, we propose to condition both the detector and description outputs on the image, and show qualitatively and quantitatively that this can improve grounding. We validate our method on image captioning, perform thorough analyses of each component and importance of the pre-trained multi-modal model, and demonstrate significant improvements over the current state of the art, specifically +7.5% in CIDEr and +1.3% in BLEU-4 metrics.