Beyond Generation: Multi-Hop Reasoning for Factual Accuracy in Vision-Language Models

TL;DR

This paper presents a framework for multi-hop reasoning in vision-language models using structured knowledge graphs, significantly improving factual accuracy in image captioning tasks by enabling systematic reasoning across multiple steps.

Contribution

It introduces a novel knowledge-guided reasoning framework for VLMs that leverages structured knowledge graphs to enhance factual accuracy through multi-hop verification.

Findings

Improves factual accuracy by approximately 31% on a curated dataset.

Demonstrates effectiveness of multi-hop reasoning in multimodal systems.

Analyzes reasoning patterns and failure modes in VLMs.

Abstract

Visual Language Models (VLMs) are powerful generative tools but often produce factually inaccurate outputs due to a lack of robust reasoning capabilities. While extensive research has been conducted on integrating external knowledge for reasoning in large language models (LLMs), such efforts remain underexplored in VLMs, where the challenge is compounded by the need to bridge multiple modalities seamlessly. This work introduces a framework for knowledge-guided reasoning in VLMs, leveraging structured knowledge graphs for multi-hop verification using image-captioning task to illustrate our framework. Our approach enables systematic reasoning across multiple steps, including visual entity recognition, knowledge graph traversal, and fact-based caption refinement. We evaluate the framework using hierarchical, triple-based and bullet-point based knowledge representations, analyzing their effectiveness in factual accuracy and logical inference. Empirical results show that our approach improves factual accuracy by approximately 31% on preliminary experiments on a curated dataset of mixtures from Google Landmarks v2, Conceptual captions and Coco captions revealing key insights into reasoning patterns and failure modes. This work demonstrates the potential of integrating external knowledge for advancing reasoning in VLMs, paving the way for more reliable and knowledgable multimodal systems.