FoCTTA: Low-Memory Continual Test-Time Adaptation with Focus

TL;DR

FoCTTA introduces a memory-efficient test-time adaptation method that selectively updates critical representation layers, significantly reducing memory usage while improving adaptation accuracy for IoT applications under domain shifts.

Contribution

It proposes a novel approach to identify and adapt only drift-sensitive layers, eliminating the need for large batch sizes and extensive activation storage, thus making CTTA practical for IoT devices.

Findings

Improves adaptation accuracy by up to 14.8% on benchmark datasets.

Reduces memory usage by approximately three times across various batch sizes.

Maintains effective adaptation with significantly lower memory requirements.

Abstract

Continual adaptation to domain shifts at test time (CTTA) is crucial for enhancing the intelligence of deep learning enabled IoT applications. However, prevailing TTA methods, which typically update all batch normalization (BN) layers, exhibit two memory inefficiencies. First, the reliance on BN layers for adaptation necessitates large batch sizes, leading to high memory usage. Second, updating all BN layers requires storing the activations of all BN layers for backpropagation, exacerbating the memory demand. Both factors lead to substantial memory costs, making existing solutions impractical for IoT devices. In this paper, we present FoCTTA, a low-memory CTTA strategy. The key is to automatically identify and adapt a few drift-sensitive representation layers, rather than blindly update all BN layers. The shift from BN to representation layers eliminates the need for large batch sizes. Also, by updating adaptation-critical layers only, FoCTTA avoids storing excessive activations. This focused adaptation approach ensures that FoCTTA is not only memory-efficient but also maintains effective adaptation. Evaluations show that FoCTTA improves the adaptation accuracy over the state-of-the-arts by 4.5%, 4.9%, and 14.8% on CIFAR10-C, CIFAR100-C, and ImageNet-C under the same memory constraints. Across various batch sizes, FoCTTA reduces the memory usage by 3-fold on average, while improving the accuracy by 8.1%, 3.6%, and 0.2%, respectively, on the three datasets.