SATA: Spatial Autocorrelation Token Analysis for Enhancing the Robustness of Vision Transformers

TL;DR

SATA introduces a novel method that leverages spatial autocorrelation between tokens to improve the robustness and efficiency of vision transformers without retraining, achieving state-of-the-art results on multiple benchmarks.

Contribution

The paper proposes SATA, a new technique that enhances ViT robustness by analyzing token spatial relationships, integrated without retraining, and reduces computational load.

Findings

Achieves 94.9% top-1 accuracy on ImageNet-1K.

Sets new state-of-the-art robustness benchmarks.

Improves efficiency by reducing FFN computational load.

Abstract

Over the past few years, vision transformers (ViTs) have consistently demonstrated remarkable performance across various visual recognition tasks. However, attempts to enhance their robustness have yielded limited success, mainly focusing on different training strategies, input patch augmentation, or network structural enhancements. These approaches often involve extensive training and fine-tuning, which are time-consuming and resource-intensive. To tackle these obstacles, we introduce a novel approach named Spatial Autocorrelation Token Analysis (SATA). By harnessing spatial relationships between token features, SATA enhances both the representational capacity and robustness of ViT models. This is achieved through the analysis and grouping of tokens according to their spatial autocorrelation scores prior to their input into the Feed-Forward Network (FFN) block of the self-attention mechanism. Importantly, SATA seamlessly integrates into existing pre-trained ViT baselines without requiring retraining or additional fine-tuning, while concurrently improving efficiency by reducing the computational load of the FFN units. Experimental results show that the baseline ViTs enhanced with SATA not only achieve a new state-of-the-art top-1 accuracy on ImageNet-1K image classification (94.9%) but also establish new state-of-the-art performance across multiple robustness benchmarks, including ImageNet-A (top-1=63.6%), ImageNet-R (top-1=79.2%), and ImageNet-C (mCE=13.6%), all without requiring additional training or fine-tuning of baseline models.