Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation

TL;DR

PACE is a novel backpropagation-free system for continual test-time adaptation that optimizes normalization parameters efficiently, achieving state-of-the-art results with over 50% reduced runtime.

Contribution

It introduces a covariance matrix adaptation strategy with a low-dimensional subspace for high-dimensional parameter optimization in a backpropagation-free setting.

Findings

Achieves state-of-the-art accuracy on multiple benchmarks.

Reduces runtime by over 50% compared to existing methods.

Effectively adapts to continual distribution shifts.

Abstract

We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters of normalization layers. Existing derivative-free approaches struggle to balance runtime efficiency with learning capacity, as they either restrict updates to input prompts or require continuous, resource-intensive adaptation regardless of domain stability. To address these limitations, PACE leverages the Covariance Matrix Adaptation Evolution Strategy with the Fastfood projection to optimize high-dimensional affine parameters within a low-dimensional subspace, leading to superior adaptive performance. Furthermore, we enhance the runtime efficiency by incorporating an adaptation stopping criterion and a domain-specialized vector bank to eliminate redundant computation. Our framework achieves state-of-the-art accuracy across multiple benchmarks under continual distribution shifts, reducing runtime by over 50% compared to existing backpropagation-free methods.