Multimodal RAG Enhanced Visual Description

TL;DR

This paper introduces a lightweight, training-free retrieval-augmented generation method that aligns visual and textual modalities in large multimodal models, improving image description quality without extensive fine-tuning.

Contribution

It proposes a novel, efficient approach using a linear mapping for modality alignment in large multimodal models, avoiding costly training and fine-tuning.

Findings

Significant improvements on benchmark datasets

Effective modality gap bridging without training

Enhanced image description generation quality

Abstract

Textual descriptions for multimodal inputs entail recurrent refinement of queries to produce relevant output images. Despite efforts to address challenges such as scaling model size and data volume, the cost associated with pre-training and fine-tuning remains substantial. However, pre-trained large multimodal models (LMMs) encounter a modality gap, characterised by a misalignment between textual and visual representations within a common embedding space. Although fine-tuning can potentially mitigate this gap, it is typically expensive and impractical due to the requirement for extensive domain-driven data. To overcome this challenge, we propose a lightweight training-free approach utilising Retrieval-Augmented Generation (RAG) to extend across the modality using a linear mapping, which can be computed efficiently. During inference, this mapping is applied to images embedded by an LMM enabling retrieval of closest textual descriptions from the training set. These textual descriptions, in conjunction with an instruction, cater as an input prompt for the language model to generate new textual descriptions. In addition, we introduce an iterative technique for distilling the mapping by generating synthetic descriptions via the language model facilitating optimisation for standard utilised image description measures. Experimental results on two benchmark multimodal datasets demonstrate significant improvements.