ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model

TL;DR

This paper systematically investigates design choices in visual grounding for multimodal large language models, providing insights and optimizations that significantly improve performance on key benchmarks.

Contribution

It offers a comprehensive analysis of visual grounding paradigms and data design choices, guiding better fine-tuning strategies for MLLMs in VG tasks.

Findings

Identifies the most effective visual grounding paradigm.

Provides ablation results on grounding data design.

Achieves up to +7.0% improvement on RefCOCOg.

Abstract

Fine-grained multimodal capability in Multimodal Large Language Models (MLLMs) has emerged as a critical research direction, particularly for tackling the visual grounding (VG) problem. Despite the strong performance achieved by existing approaches, they often employ disparate design choices when fine-tuning MLLMs for VG, lacking systematic verification to support these designs. To bridge this gap, this paper presents a comprehensive study of various design choices that impact the VG performance of MLLMs. We conduct our analysis using LLaVA-1.5, which has been widely adopted in prior empirical studies of MLLMs. While more recent models exist, we follow this convention to ensure our findings remain broadly applicable and extendable to other architectures. We cover two key aspects: (1) exploring different visual grounding paradigms in MLLMs, identifying the most effective design, and providing our insights; and (2) conducting ablation studies on the design of grounding data to optimize MLLMs' fine-tuning for the VG task. Finally, our findings contribute to a stronger MLLM for VG, achieving improvements of +5.6% / +6.9% / +7.0% on RefCOCO/+/g over the LLaVA-1.5.