When to Think and When to Look: Uncertainty-Guided Lookback

TL;DR

This paper analyzes how explicit reasoning chains affect visual language model performance, revealing that strategic, image-referential lookback improves reasoning accuracy and proposing an uncertainty-guided decoding method that enhances results across multiple benchmarks.

Contribution

It provides the first systematic analysis of reasoning in LVLMs, identifies effective short lookback strategies, and introduces an uncertainty-guided decoding approach that improves visual reasoning performance.

Findings

Long reasoning chains can lead to errors and ignore images.

Short, image-referential lookback phrases improve reasoning accuracy.

Uncertainty-guided lookback enhances performance across multiple benchmarks.

Abstract

Test-time thinking (that is, generating explicit intermediate reasoning chains) is known to boost performance in large language models and has recently shown strong gains for large vision language models (LVLMs). However, despite these promising results, there is still no systematic analysis of how thinking actually affects visual reasoning. We provide the first such analysis with a large scale, controlled comparison of thinking for LVLMs, evaluating ten variants from the InternVL3.5 and Qwen3-VL families on MMMU-val under generous token budgets and multi pass decoding. We show that more thinking is not always better; long chains often yield long wrong trajectories that ignore the image and underperform the same models run in standard instruct mode. A deeper analysis reveals that certain short lookback phrases, which explicitly refer back to the image, are strongly enriched in successful trajectories and correlate with better visual grounding. Building on this insight, we propose uncertainty guided lookback, a training free decoding strategy that combines an uncertainty signal with adaptive lookback prompts and breadth search. Our method improves overall MMMU performance, delivers the largest gains in categories where standard thinking is weak, and outperforms several strong decoding baselines, setting a new state of the art under fixed model families and token budgets. We further show that this decoding strategy generalizes, yielding consistent improvements on five additional benchmarks, including two broad multimodal suites and math focused visual reasoning datasets.