Contrastive Residual Energy Test-time Adaptation

TL;DR

CreTTA introduces a contrastive residual energy approach for scalable, reliable test-time adaptation that avoids costly sampling and overfitting, improving model calibration in real-world scenarios.

Contribution

It reformulates marginal distribution adaptation as learning a residual energy function with a contrastive objective, eliminating sampling and reducing overfitting.

Findings

Achieves scalable and well-calibrated adaptation in real-world settings.

Removes the need for costly sampling by canceling the partition function.

Prevents overfitting through adaptive gradient reweighting.

Abstract

Test-time adaptation (TTA) enhances model robustness by enabling adaptation to target distributions that differ from training distributions, improving real-world generalizability. However, most existing TTA approaches focus on adjusting the conditional distribution and therefore exhibit poor calibration, as they rely on uncertain predictions in the absence of labels. Energy-based TTA frameworks provide an alternative by modeling the marginal distribution of target data without depending on label predictions, but their reliance on costly sampling hinders scalability in real-world scenarios where decisions must be made without latency. In this work, we propose Contrastive Residual Energy Test-time Adaptation (CreTTA), a practical solution for reliable adaptation. We theoretically reformulate the marginal distribution adaptation as learning a residual energy function. This formulation leads to a contrastive objective where the intractable partition function mathematically cancels out, removing sampling and approximation error.Crucially, our analysis reveals that this design prevents overfitting through an adaptive gradient reweighting mechanism that leverages relative energy differences, avoiding the self-confirming bias of entropy minimization. Extensive experiments demonstrate that CreTTA achieves scalable and well-calibrated adaptation under real-world computational constraints.