Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering

TL;DR

This paper introduces RMCD, a relevance-aware decoding method for retrieval-augmented visual question answering, which effectively leverages multiple contexts to improve accuracy without additional training.

Contribution

The paper proposes a novel decoding approach that weights multiple contexts by relevance, enhancing retrieval-augmented VQA performance without extra training.

Findings

RMCD outperforms existing decoding methods on three benchmarks.

RMCD is robust to retrieval quality variations.

RMCD can be applied without additional training.

Abstract

Despite the remarkable capabilities of Large Vision Language Models (LVLMs), they still lack detailed knowledge about specific entities. Retrieval-augmented Generation (RAG) is a widely adopted solution that enhances LVLMs by providing additional contexts from an external Knowledge Base. However, we observe that previous decoding methods for RAG are sub-optimal as they fail to sufficiently leverage multiple relevant contexts and suppress the negative effects of irrelevant contexts. To this end, we propose Relevance-aware Multi-context Contrastive Decoding (RMCD), a novel decoding method for RAG. RMCD outputs a final prediction by combining outputs predicted with each context, where each output is weighted based on its relevance to the question. By doing so, RMCD effectively aggregates useful information from multiple relevant contexts while also counteracting the negative effects of irrelevant ones. Experiments show that RMCD consistently outperforms other decoding methods across multiple LVLMs, achieving the best performance on three knowledge-intensive visual question-answering benchmarks. Also, RMCD can be simply applied by replacing the decoding method of LVLMs without additional training. Analyses also show that RMCD is robust to the retrieval results, consistently performing the best across the weakest to the strongest retrieval results. Code is available at https://github.com/mlvlab/RMCD.