Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models

TL;DR

The paper introduces Chain-of-Spot, an interactive reasoning method that enhances large vision-language models by focusing on key image regions, leading to improved visual understanding and reasoning without increasing image resolution.

Contribution

It presents a novel interactive reasoning approach that improves feature extraction in LVLMs by focusing on key regions, achieving state-of-the-art results.

Findings

Significant performance improvements across multiple benchmarks.

Enhanced visual reasoning without increasing image resolution.

Effective integration with existing instruct-following models.

Abstract

In the realm of vision-language understanding, the proficiency of models in interpreting and reasoning over visual content has become a cornerstone for numerous applications. However, it is challenging for the visual encoder in Large Vision-Language Models (LVLMs) to extract useful features tailored to questions that aid the language model's response. Furthermore, a common practice among existing LVLMs is to utilize lower-resolution images, which restricts the ability for visual recognition. Our work introduces the Chain-of-Spot (CoS) method, which we describe as Interactive Reasoning, a novel approach that enhances feature extraction by focusing on key regions of interest (ROI) within the image, corresponding to the posed questions or instructions. This technique allows LVLMs to access more detailed visual information without altering the original image resolution, thereby offering multi-granularity image features. By integrating Chain-of-Spot with instruct-following LLaVA-1.5 models, the process of image reasoning consistently improves performance across a wide range of multimodal datasets and benchmarks without bells and whistles and achieves new state-of-the-art results. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content, paving the way for more sophisticated visual instruction-following applications. Code and models are available at https://github.com/dongyh20/Chain-of-Spot