On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning

TL;DR

This paper investigates the realistic risks of test-time data poisoning attacks on test-time adaptation methods, proposing new attack strategies and analyzing defenses, revealing that TTA methods are more robust than previously thought.

Contribution

The study introduces a realistic attack method for TTA, evaluates existing attacks and defenses, and provides insights into the robustness of TTA against data poisoning.

Findings

TTA methods are more robust than previously believed.

Proposed attack method effectively poisons without access to benign data.

Analysis of defense strategies for adversarial robustness in TTA.

Abstract

Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack order, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods. The source code is available at https://github.com/Gorilla-Lab-SCUT/RTTDP.