Test-Time Consistency in Vision Language Models

TL;DR

This paper introduces a simple, post-hoc test-time framework to improve the semantic consistency of vision-language models without retraining, significantly enhancing their reliability across equivalent inputs.

Contribution

It proposes a novel, model-agnostic, test-time consistency method using two objectives, applicable to any VLM, to improve semantic consistency without supervised re-training.

Findings

Significant improvements in consistency on MM-R3 benchmark.

Applicable to various state-of-the-art VLMs without retraining.

Establishes a new inference-time adaptation approach for multimodal models.

Abstract

Vision-Language Models (VLMs) have achieved impressive performance across a wide range of multimodal tasks, yet they often exhibit inconsistent behavior when faced with semantically equivalent inputs, undermining their reliability and robustness. Recent benchmarks, such as MM-R3, highlight that even state-of-the-art VLMs can produce divergent predictions across semantically equivalent inputs, despite maintaining high average accuracy. Prior work addresses this issue by modifying model architectures or conducting large-scale fine-tuning on curated datasets. In contrast, we propose a simple and effective test-time consistency framework that enhances semantic consistency without supervised re-training. Our method is entirely post-hoc, model-agnostic, and applicable to any VLM with access to its weights. Given a single test point, we enforce consistent predictions via two complementary objectives: (i) a Cross-Entropy Agreement Loss that aligns predictive distributions across semantically equivalent inputs, and (ii) a Pseudo-Label Consistency Loss that draws outputs toward a self-averaged consensus. Our method is plug-and-play and leverages information from a single test input itself to improve consistency. Experiments on the MM-R3 benchmark show that our framework yields substantial gains in consistency across state-of-the-art models, establishing a new direction for inference-time adaptation in multimodal learning.