Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models

TL;DR

This paper shows that models maintaining temporally grounded beliefs during long-horizon vision-language tasks generalize better out-of-distribution, with step-level grounding quality serving as a key predictor of robustness.

Contribution

It introduces the Step Grounding Rate (SGR) as a measurable indicator of behavioral faithfulness and demonstrates its strong correlation with out-of-distribution robustness across multiple models and benchmarks.

Findings

SGR predicts out-of-distribution retention with r=0.83

Grounding quality varies significantly within parameter-matched models

Counterfactual and cross-architecture tests confirm the importance of visual reliance

Abstract

We uncover a behavioral law of long-horizon vision-language models: models that maintain temporally grounded beliefs generalize better. Standard benchmarks measure only final-answer accuracy, which obscures how models use visual information; a model can guess correctly while its step-by-step reasoning is entirely unanchored to the visual input. We formalize this as behavioral faithfulness over long horizons, an empirically measurable property that quantifies whether a model's intermediate reasoning remains consistent with the evolving visual state. Across eight models on three long-horizon benchmarks, we demonstrate that temporal grounding quality is a leading indicator of robustness: the Step Grounding Rate (SGR) predicts out-of-distribution retention with $r = 0.83$ (permutation test $p = 0.003$), a relationship that holds within capacity-matched models and cannot be explained by scale or in-distribution accuracy. Critically, grounding quality varies by up to 10.8 percentage points within parameter-matched 7B models despite similar accuracy, revealing it as an independent axis of model capability. Multiple robustness checks confirm the signal reflects genuine visual reliance: counterfactual traces drop SGR by 26--41 percentage points, cross-architecture verifiers agree at $\rho = 0.96$, random reasoning scores near chance ($\sim 18\%$), and the predictor remains strong even without explicit reasoning disclosure ($r = 0.78$).