GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

TL;DR

This paper introduces a systematic benchmark for memory policies in test-time adaptation, emphasizing the importance of intra-class diversity for effective online model adaptation under various data stream conditions.

Contribution

It decouples memory design from adaptation algorithms, revealing diversity as key for robust TTA, and proposes the GOTTA family of diversity-aware memory policies as effective replacements.

Findings

Diversity-aware memory improves adaptation under constrained memory and non-i.i.d. streams.

Intra-class diversity prevents redundant buffers and maintains representative signals.

Memory management is crucial for robust test-time adaptation.

Abstract

Test-time adaptation (TTA) enables a pre-trained model to adapt online to an unlabeled test stream under distribution shift. While most TTA research focuses on the adaptation objective, practical streams also depend critically on the memory used to select which test samples drive adaptation. Existing memory mechanisms are usually evaluated as components of specific TTA algorithms, making it difficult to isolate which memory design choices matter and when they matter. In this work, we provide a systematic benchmark that decouples memory from the adaptation algorithm and evaluates memory policies under unified conditions across i.i.d., non-i.i.d., continual, and practical test streams. Our study shows that effective memory management requires more than retaining recent or class-balanced samples. In particular, intra-class diversity is a key factor for avoiding redundant buffers and maintaining representative adaptation signals under temporally correlated and label-skewed streams. Motivated by this finding, we introduce Guided Observational Test-Time Adaptation (GOTTA), a family of diversity-aware memory policies that combine class-balanced allocation with feature-space diversity. GOTTA memories act as drop-in replacements for existing buffers and can be paired with different TTA objectives. Across corruption benchmarks and video-stream settings, diversity-aware memory improves adaptation most clearly under constrained memory budgets and challenging non-i.i.d. streams, while remaining competitive as memory capacity increases. These results highlight memory management as a first-class component of robust test-time adaptation and identify diversity as a central principle for practical TTA.