Cross-Modal Retrieval Augmentation for Multi-Modal Classification

TL;DR

This paper introduces a retrieval-augmented multi-modal approach that aligns images and captions in a shared space, significantly enhancing visual question answering performance by leveraging external knowledge sources and novel inference techniques.

Contribution

The paper presents a new alignment model for images and captions and demonstrates how retrieval-augmented transformers improve VQA results over existing baselines.

Findings

Improved image-caption retrieval performance

Enhanced VQA accuracy with retrieval-augmented models

Effective inference-time hot-swapping of indices

Abstract

Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model improve results on VQA over strong baselines. We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.