EVA-0: Test-Time Model Evolution with Only Two Forward Passes per Sample

TL;DR

EVA-0 introduces a highly efficient test-time model adaptation method that requires only two forward passes per sample, eliminating backpropagation and enabling deployment on resource-constrained devices.

Contribution

The paper presents EVA-0, a zeroth-order adaptation framework that overcomes key obstacles in test-time optimization without backpropagation, suitable for edge deployment.

Findings

EVA-0 outperforms BP-based methods like DeYO on ImageNet-C.

EVA-0 achieves a 14x speed-up over FOA.

EVA-0 maintains high accuracy with only two forward passes per sample.

Abstract

Test-time model evolution offers a promising way for deployed models to improve from unlabeled test-time experience, yet most existing methods depend on backpropagation (BP), which incurs substantial memory overhead and makes them difficult to deploy on edge devices, quantized models, specialized accelerators, or black-box models. In this work, we study test-time model evolution under a strict two-forward budget, a setting that pushes adaptation toward highly efficient real-world deployment. We reveal three key obstacles in zeroth-order test-time optimization: susceptibility to shortcut solutions, uncontrolled weight drift, and ineffective update direction estimation. To overcome them, we propose EVA-0, a minimal zeroth-order adaptation framework that: 1) keeps the loss scale-invariant to prevent shortcut solutions; 2) devises an anchor-guided optimization strategy to alleviate weight drift; 3) uses sample-wise symmetric two-sided perturbation for update direction estimation and inference. EVA-0 requires no BP and performs both inference and adaptation within only two forward passes per sample. Results on ImageNet-C & ViT-Base show that EVA-0 outperforms both BP-based DeYO and BP-free FOA, while achieving a 14x speed-up over FOA. Code will be released.