v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning

TL;DR

The paper introduces v1, a model extension enabling active visual referencing by selecting and copying relevant image patches during multimodal reasoning, improving focus and performance.

Contribution

It proposes a novel point-and-copy mechanism for visual grounding in multimodal reasoning models, trained on a large dataset, enhancing interpretability and accuracy.

Findings

v1 outperforms baselines on multimodal reasoning benchmarks.

The point-and-copy mechanism maintains alignment between visual evidence and reasoning.

Training on v1g dataset enables effective learning of visual referencing.

Abstract

When thinking with images, humans rarely rely on a single glance: they revisit visual evidence while reasoning. In contrast, most Multimodal Language Models encode an image once to key-value cache and then reason purely in text, making it hard to re-ground intermediate steps. We empirically confirm this: as reasoning chains lengthen, models progressively lose focus on relevant regions. We introduce v1, a lightweight extension for active visual referencing via point-and-copy: the model selects relevant image patches and copies their embeddings back into the reasoning stream. Crucially, our point-and-copy mechanism retrieves patches using their semantic representations as keys, ensuring perceptual evidence remains aligned with the reasoning space. To train this behavior, we build v1g, a dataset of 300K multimodal reasoning traces with interleaved grounding annotations. Across multimodal mathematical reasoning benchmarks, v1 consistently outperforms comparable baselines.