Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering

TL;DR

This paper introduces a multimodal retrieval-augmented generation framework with scene graphs to improve object recognition, spatial understanding, and accuracy in visual question answering, especially in complex scenes.

Contribution

The novel framework integrates structured scene graphs into MLLMs, enhancing their ability to recognize objects and understand spatial relationships in challenging visual contexts.

Findings

Outperforms existing MLLMs in VQA accuracy

Improves object localization and counting in complex scenes

Achieves better results on VG-150 and AUG datasets

Abstract

Multimodal large language models (MLLMs), such as GPT-4o, Gemini, LLaVA, and Flamingo, have made significant progress in integrating visual and textual modalities, excelling in tasks like visual question answering (VQA), image captioning, and content retrieval. They can generate coherent and contextually relevant descriptions of images. However, they still face challenges in accurately identifying and counting objects and determining their spatial locations, particularly in complex scenes with overlapping or small objects. To address these limitations, we propose a novel framework based on multimodal retrieval-augmented generation (RAG), which introduces structured scene graphs to enhance object recognition, relationship identification, and spatial understanding within images. Our framework improves the MLLM's capacity to handle tasks requiring precise visual descriptions, especially in scenarios with challenging perspectives, such as aerial views or scenes with dense object arrangements. Finally, we conduct extensive experiments on the VG-150 dataset that focuses on first-person visual understanding and the AUG dataset that involves aerial imagery. The results show that our approach consistently outperforms existing MLLMs in VQA tasks, which stands out in recognizing, localizing, and quantifying objects in different spatial contexts and provides more accurate visual descriptions.