Test-Time Hinting for Black-Box Vision-Language Models

TL;DR

This paper introduces Test-Time Hinting, a black-box method that enhances vision-language model accuracy using a single API call by predicting effective hints to mitigate common failure modes.

Contribution

It presents a novel, efficient approach for improving VLM performance via a lightweight hint generator that requires only black-box access and generalizes across models and benchmarks.

Findings

Test-Time Hinting improves VLM accuracy on natural-image VQA benchmarks.

The method generalizes to unseen benchmarks and models without retraining.

It achieves these gains with a single model call and minimal computational overhead.

Abstract

Test-time scaling (TTS) methods have proven highly effective for LLMs, yet their application to vision-language models (VLMs) remains relatively underexplored. Existing VLM TTS methods largely require open-weight model access or expensive repeated sampling, and are evaluated primarily on multimodal mathematical and scientific reasoning benchmarks rather than general visual understanding tasks. In this paper, we propose Test-Time Hinting, a method that improves VLM performance via a single VLM call and requiring only black-box API access, which makes it broadly applicable to frontier closed-weight models. Our method is motivated by the observation that VLM errors tend to cluster around recurring failure patterns. We therefore train a lightweight hint generator model to predict, for a given test input, which "hint" should be prepended to the prompt, providing targeted contextual or procedural guidance that steers the VLM away from its characteristic failure modes. We show that Test-Time Hinting improves the accuracy of multiple closed-weight VLMs on natural-image VQA benchmarks and that these gains generalize to unseen benchmarks and VLMs without retraining the hint generator.