TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models

TL;DR

This paper introduces Test-Time Padding (TTP), a lightweight method for detecting and adapting to adversarial attacks on vision-language models like CLIP, improving robustness without sacrificing accuracy.

Contribution

TTP is a novel test-time framework that detects adversarial inputs via feature similarity shifts and employs targeted padding-based adaptation, outperforming existing defenses.

Findings

TTP reliably detects adversarial inputs across models and datasets.

TTP improves robustness without reducing clean accuracy.

Experimental results surpass state-of-the-art defenses.

Abstract

Vision-Language Models (VLMs), such as CLIP, have achieved impressive zero-shot recognition performance but remain highly susceptible to adversarial perturbations, posing significant risks in safety-critical scenarios. Previous training-time defenses rely on adversarial fine-tuning, which requires labeled data and costly retraining, while existing test-time strategies fail to reliably distinguish between clean and adversarial inputs, thereby preventing both adversarial robustness and clean accuracy from reaching their optimum. To address these limitations, we propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference. TTP identifies adversarial inputs via the cosine similarity shift between CLIP feature embeddings computed before and after spatial padding, yielding a universal threshold for reliable detection across architectures and datasets. For detected adversarial cases, TTP employs trainable padding to restore disrupted attention patterns, coupled with a similarity-aware ensemble strategy for a more robust final prediction. For clean inputs, TTP leaves them unchanged by default or optionally integrates existing test-time adaptation techniques for further accuracy gains. Comprehensive experiments on diverse CLIP backbones and fine-grained benchmarks show that TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy. The code for this paper will be released soon.