FedPoisonTTP: A Threat Model and Poisoning Attack for Federated Test-Time Personalization

TL;DR

This paper introduces FedPoisonTTP, a grey-box poisoning attack framework targeting federated test-time personalization, demonstrating how adversaries can significantly degrade model performance through crafted poisoned inputs during local adaptation.

Contribution

The paper presents a novel poisoning attack method specifically designed for federated test-time personalization, highlighting security vulnerabilities and attack strategies in this setting.

Findings

Poisoned inputs can substantially reduce model performance.

Adversarial updates spread and amplify the attack impact.

The attack remains effective against common defense mechanisms.

Abstract

Test-time personalization in federated learning enables models at clients to adjust online to local domain shifts, enhancing robustness and personalization in deployment. Yet, existing federated learning work largely overlooks the security risks that arise when local adaptation occurs at test time. Heterogeneous domain arrivals, diverse adaptation algorithms, and limited cross-client visibility create vulnerabilities where compromised participants can craft poisoned inputs and submit adversarial updates that undermine both global and per-client performance. To address this threat, we introduce FedPoisonTTP, a realistic grey-box attack framework that explores test-time data poisoning in the federated adaptation setting. FedPoisonTTP distills a surrogate model from adversarial queries, synthesizes in-distribution poisons using feature-consistency, and optimizes attack objectives to generate high-entropy or class-confident poisons that evade common adaptation filters. These poisons are injected during local adaptation and spread through collaborative updates, leading to broad degradation. Extensive experiments on corrupted vision benchmarks show that compromised participants can substantially diminish overall test-time performance.