Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models

TL;DR

This paper presents PromViL, a hierarchical framework that improves large vision-language models' ability to perform grounded compositional reasoning by progressively aligning multi-modal concepts from simple to complex.

Contribution

Introduction of a hierarchical multi-granular alignment framework and a novel dataset for enhancing compositional visual reasoning in LVLMs.

Findings

Significant improvements on visual grounding tasks

Enhanced performance on compositional question answering

Effective hierarchical alignment strategy

Abstract

Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between entities. This paper introduces Progressive multi-granular Vision-Language alignments (PromViL), a novel framework to enhance LVLMs' ability in performing grounded compositional visual reasoning tasks. Our approach constructs a hierarchical structure of multi-modal alignments, ranging from simple to complex concepts. By progressively aligning textual descriptions with corresponding visual regions, our model learns to leverage contextual information from lower levels to inform higher-level reasoning. To facilitate this learning process, we introduce a data generation process that creates a novel dataset derived from Visual Genome, providing a wide range of nested compositional vision-language pairs. Experimental results demonstrate that our PromViL framework significantly outperforms baselines on various visual grounding and compositional question answering tasks. The code is available at: https://github.com/lqh52/PromViL.