PatchCue: Enhancing Vision-Language Model Reasoning with Patch-Based Visual Cues

TL;DR

PatchCue introduces a patch-based visual cue method that enhances vision-language models' reasoning by aligning visual cues with human perception, leading to improved performance across various multimodal understanding tasks.

Contribution

The paper proposes a novel patch-based visual cue paradigm and a two-stage training approach to significantly improve visual reasoning in VLMs.

Findings

PatchCue outperforms pixel-level cues in experiments.

Enhanced reasoning performance across multiple benchmarks.

Better alignment with human perceptual habits.

Abstract

Vision-Language Models (VLMs) have achieved remarkable progress on a wide range of challenging multimodal understanding and reasoning tasks. However, existing reasoning paradigms, such as the classical Chain-of-Thought (CoT), rely solely on textual information and often underutilize important visual cues. While prior work has incorporated pixel-level visual cues, these representations require precise spatial localization, introducing additional learning complexity. To address this, we propose PatchCue, a novel patch-based visual cue paradigm designed to significantly enhance the visual reasoning capabilities of VLMs. By partitioning images into patches and representing cues at the patch level, PatchCue aligns better with human perceptual habits and leverages the patch-tokenized input of modern VLMs. We train VLMs using a two-stage approach: cold-start supervised fine-tuning to output patch-level cues, followed by reinforcement learning with a process-supervised cue reward that guides intermediate visual reasoning steps. Extensive experiments on multiple VLMs and diverse benchmarks, including general visual question answering, complex reasoning, and document understanding, demonstrate that PatchCue consistently improves overall model performance. Our results show that patch-level cues outperform both pixel-level bounding boxes and point-based cues, providing a more effective and cognitively aligned visual reasoning paradigm.