Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models

TL;DR

This paper introduces BETA, a novel framework for efficient and stable test-time adaptation of black-box models that requires no additional API calls and maintains real-time inference speed.

Contribution

BETA employs a lightweight white-box model to enable gradient-based adaptation in black-box settings without extra API calls or latency.

Findings

BETA improves ImageNet-C accuracy by +7.1% on ViT-B/16.

BETA achieves +3.4% accuracy on CLIP.

BETA matches ZOO performance at 250x lower cost.

Abstract

Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains a largely unexplored challenge. Existing approaches such as post-hoc output refinement offer limited adaptive capacity, while Zeroth-Order Optimization (ZOO) enables input-space adaptation but faces high query costs and optimization challenges in the unsupervised TTA setting. We introduce BETA (Black-box Efficient Test-time Adaptation), a framework that addresses these limitations by employing a lightweight, local white-box steering model to create a tractable gradient pathway. Through a prediction harmonization technique combined with consistency regularization and prompt learning-oriented filtering, BETA enables stable adaptation with no additional API calls and negligible latency beyond standard inference. On ImageNet-C, BETA achieves a +7.1% accuracy gain on ViT-B/16 and +3.4% on CLIP, surpassing strong white-box and gray-box methods including TENT and TPT. On a commercial API, BETA achieves comparable performance to ZOO at 250x lower cost while maintaining real-time inference speed, establishing it as a practical and efficient solution for real-world black-box TTA.