GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models

TL;DR

This paper introduces GeM-VG, a multimodal large language model designed for generalized multi-image visual grounding, supported by a new dataset and a hybrid finetuning strategy, achieving superior performance across various tasks.

Contribution

It presents a unified model for diverse multi-image grounding tasks, introduces the MG-Data-240K dataset, and proposes a hybrid reinforcement finetuning method to enhance reasoning and perception.

Findings

Outperforms previous MLLMs on MIG-Bench and MC-Bench by 2.0% and 9.7%.

Achieves 9.1% improvement on ODINW for single-image grounding.

Demonstrates strong multi-image understanding capabilities.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive progress in single-image grounding and general multi-image understanding. Recently, some methods begin to address multi-image grounding. However, they are constrained by single-target localization and limited types of practical tasks, due to the lack of unified modeling for generalized grounding tasks. Therefore, we propose GeM-VG, an MLLM capable of Generalized Multi-image Visual Grounding. To support this, we systematically categorize and organize existing multi-image grounding tasks according to their reliance of cross-image cues and reasoning, and introduce the MG-Data-240K dataset, addressing the limitations of existing datasets regarding target quantity and image relation. To tackle the challenges of robustly handling diverse multi-image grounding tasks, we further propose a hybrid reinforcement finetuning strategy that integrates chain-of-thought (CoT) reasoning and direct answering, considering their complementary strengths. This strategy adopts an R1-like algorithm guided by a carefully designed rule-based reward, effectively enhancing the model's overall perception and reasoning capabilities. Extensive experiments demonstrate the superior generalized grounding capabilities of our model. For multi-image grounding, it outperforms the previous leading MLLMs by 2.0% and 9.7% on MIG-Bench and MC-Bench, respectively. In single-image grounding, it achieves a 9.1% improvement over the base model on ODINW. Furthermore, our model retains strong capabilities in general multi-image understanding.