Test-Time Model Adaptation with Only Forward Passes

TL;DR

This paper introduces FOA, a test-time adaptation method that updates models using only forward passes, making it suitable for resource-limited devices by avoiding backpropagation and weight modifications.

Contribution

The paper proposes a novel derivative-free adaptation approach using prompt learning and activation shifting, enabling efficient test-time adaptation without backpropagation.

Findings

FOA outperforms gradient-based methods like TENT on quantized models.

FOA achieves up to 24-fold memory reduction on ImageNet-C.

FOA improves adaptation performance on shifted test samples.

Abstract

Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts. However, in real-world scenarios, models are usually deployed on resource-limited devices, e.g., FPGAs, and are often quantized and hard-coded with non-modifiable parameters for acceleration. In light of this, existing methods are often infeasible since they heavily depend on computation-intensive backpropagation for model updating that may be not supported. To address this, we propose a test-time Forward-Optimization Adaptation (FOA) method. In FOA, we seek to solely learn a newly added prompt (as model's input) via a derivative-free covariance matrix adaptation evolution strategy. To make this strategy work stably under our online unsupervised setting, we devise a novel fitness function by measuring test-training statistic discrepancy and model prediction entropy. Moreover, we design an activation shifting scheme that directly tunes the model activations for shifted test samples, making them align with the source training domain, thereby further enhancing adaptation performance. Without using any backpropagation and altering model weights, FOA runs on quantized 8-bit ViT outperforms gradient-based TENT on full-precision 32-bit ViT, while achieving an up to 24-fold memory reduction on ImageNet-C.