Limits and Gains of Test-Time Scaling in Vision-Language Reasoning

TL;DR

This paper systematically evaluates test-time scaling in vision-language models, revealing its variable effectiveness depending on model type, task, and dataset, and highlighting the need for adaptive strategies.

Contribution

It provides the first comprehensive empirical analysis of test-time scaling in vision-language models across diverse benchmarks and model types.

Findings

Closed-source models benefit from structured reasoning and self-refinement.

Open-source models show inconsistent gains, with external verification being most reliable.

TTS improvements are dataset-dependent, aiding multi-step reasoning but limited in perception tasks.

Abstract

Test-time scaling (TTS) has emerged as a powerful paradigm for improving the reasoning ability of Large Language Models (LLMs) by allocating additional computation at inference, yet its application to multimodal systems such as Vision-Language Models (VLMs) remains underexplored. In this work, we present a systematic empirical study of inference time reasoning methods applied across both open-source and closed-source VLMs on different benchmarks. Our results reveal that while closed-source models consistently benefit from structured reasoning and iterative Self-Refinement, open-source VLMs show inconsistent behavior: external verification provides the most reliable gains, whereas iterative refinement often degrades performance. We further find that the effectiveness of TTS is dataset-dependent, yielding clear improvements on multi-step reasoning tasks but offering only limited gains on perception-focused benchmarks. These findings demonstrate that TTS is not a universal solution and must be tailored to both model capabilities and task characteristics, motivating future work on adaptive TTS strategies and multimodal reward models.