Test-Time Poisoning Attacks Against Test-Time Adaptation Models

TL;DR

This paper demonstrates that test-time adaptation models are vulnerable to poisoning attacks during deployment, significantly degrading their performance with minimal poisoned samples, highlighting the need for security-aware TTA design.

Contribution

First to analyze and demonstrate the vulnerability of mainstream TTA methods to test-time poisoning attacks using surrogate-generated poisoned samples.

Findings

TTA models can be degraded from 76.20% to 41.83% accuracy with only 10 poisoned samples.

Test-time poisoning attacks are effective against four mainstream TTA methods.

Current TTA algorithms lack sufficient security measures against poisoning threats.

Abstract

Deploying machine learning (ML) models in the wild is challenging as it suffers from distribution shifts, where the model trained on an original domain cannot generalize well to unforeseen diverse transfer domains. To address this challenge, several test-time adaptation (TTA) methods have been proposed to improve the generalization ability of the target pre-trained models under test data to cope with the shifted distribution. The success of TTA can be credited to the continuous fine-tuning of the target model according to the distributional hint from the test samples during test time. Despite being powerful, it also opens a new attack surface, i.e., test-time poisoning attacks, which are substantially different from previous poisoning attacks that occur during the training time of ML models (i.e., adversaries cannot intervene in the training process). In this paper, we perform the first test-time poisoning attack against four mainstream TTA methods, including TTT, DUA, TENT, and RPL. Concretely, we generate poisoned samples based on the surrogate models and feed them to the target TTA models. Experimental results show that the TTA methods are generally vulnerable to test-time poisoning attacks. For instance, the adversary can feed as few as 10 poisoned samples to degrade the performance of the target model from 76.20% to 41.83%. Our results demonstrate that TTA algorithms lacking a rigorous security assessment are unsuitable for deployment in real-life scenarios. As such, we advocate for the integration of defenses against test-time poisoning attacks into the design of TTA methods.