VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation

TL;DR

VLR-Bench is a new multilingual vision-language retrieval benchmark designed to evaluate and improve retrieval-augmented generation in vision-language models, featuring a dataset of 32,000 instruction-following examples.

Contribution

The paper introduces VLR-Bench and VLR-IF datasets, enabling evaluation and training of VLMs for retrieval-augmented answer generation with multiple input passages.

Findings

VLR-Bench effectively evaluates VLMs on retrieval capabilities.

VLR-IF enhances VLMs' ability to generate answers based on input passages.

State-of-the-art models show improved performance using the benchmark.

Abstract

We propose the VLR-Bench, a visual question answering (VQA) benchmark for evaluating vision language models (VLMs) based on retrieval augmented generation (RAG). Unlike existing evaluation datasets for external knowledge-based VQA, the proposed VLR-Bench includes five input passages. This allows testing of the ability to determine which passage is useful for answering a given query, a capability lacking in previous research. In this context, we constructed a dataset of 32,000 automatically generated instruction-following examples, which we denote as VLR-IF. This dataset is specifically designed to enhance the RAG capabilities of VLMs by enabling them to learn how to generate appropriate answers based on input passages. We evaluated the validity of the proposed benchmark and training data and verified its performance using the state-of-the-art Llama3-based VLM, the Llava-Llama-3 model. The proposed VLR-Bench and VLR-IF datasets are publicly available online.