Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

TL;DR

This paper introduces a Self-Critical Inference framework to enhance the robustness of vision-language models against language bias and sensitivity, using multi-round counterfactual reasoning and a dynamic benchmark for evaluation.

Contribution

The paper proposes a novel Self-Critical Inference framework with multi-round reasoning and a dynamic benchmark to improve and accurately evaluate vision-language model robustness.

Findings

SCI outperforms baseline methods on DRBench.

Increasing inference rounds enhances robustness.

Model-specific evaluation reveals varied failure modes.

Abstract

The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.