Vision Transformer Finetuning Benefits from Non-Smooth Components

TL;DR

This paper investigates how the plasticity of vision transformer components, particularly attention and feedforward layers, influences transfer learning performance, challenging the assumption that smoothness is always beneficial.

Contribution

It introduces a theoretical and experimental framework to analyze component plasticity in vision transformers, guiding better finetuning strategies.

Findings

High plasticity in attention modules improves finetuning results.

Feedforward layers with higher plasticity also enhance transfer learning.

Challenging the belief that smoothness always benefits transformer adaptation.

Abstract

The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their plasticity. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies low smoothness. We demonstrate through theoretical analysis and comprehensive experiments that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on the functional properties of transformers. The code is available at https://github.com/ambroiseodt/vit-plasticity.