Taming the Untamed: Graph-Based Knowledge Retrieval and Reasoning for MLLMs to Conquer the Unknown

TL;DR

This paper introduces a graph-based knowledge retrieval and reasoning framework for multimodal large language models, enhancing their ability to handle domain-specific tasks by integrating a multimodal knowledge graph and a multi-agent retriever.

Contribution

It presents a novel multimodal knowledge graph and a multi-agent retrieval system to improve MLLMs' reasoning in domain-specific, complex scenarios.

Findings

Significant performance improvement in knowledge retrieval and reasoning tasks.

Effective use of a multimodal knowledge graph in a visual game cognition context.

Demonstrated potential for future multimodal knowledge-augmented reasoning research.

Abstract

The real value of knowledge lies not just in its accumulation, but in its potential to be harnessed effectively to conquer the unknown. Although recent multimodal large language models (MLLMs) exhibit impressing multimodal capabilities, they often fail in rarely encountered domain-specific tasks due to limited relevant knowledge. To explore this, we adopt visual game cognition as a testbed and select Monster Hunter: World as the target to construct a multimodal knowledge graph (MH-MMKG), which incorporates multi-modalities and intricate entity relations. We also design a series of challenging queries based on MH-MMKG to evaluate the models' ability for complex knowledge retrieval and reasoning. Furthermore, we propose a multi-agent retriever that enables a model to autonomously search relevant knowledge without additional training. Experimental results show that our approach significantly enhances the performance of MLLMs, providing a new perspective on multimodal knowledge-augmented reasoning and laying a solid foundation for future research.