Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers

TL;DR

This paper identifies a variance shift issue in the positional embeddings of Vision Transformers caused by common data augmentations like Mixup, and proposes guidelines to configure augmentations for improved model stability and performance.

Contribution

It reveals the hidden impact of data augmentation on positional embeddings and provides a detailed analysis and configuration guidelines to mitigate variance shift effects in ViTs.

Findings

Configuring data augmentations improves ViT performance.

Variance shift in positional embeddings degrades test performance.

Guidelines help stabilize positional embeddings during training.

Abstract

Vision transformers (ViTs) have demonstrated remarkable performance in a variety of vision tasks. Despite their promising capabilities, training a ViT requires a large amount of diverse data. Several studies empirically found that using rich data augmentations, such as Mixup, Cutmix, and random erasing, is critical to the successful training of ViTs. Now, the use of rich data augmentations has become a standard practice in the current state. However, we report a vulnerability to this practice: Certain data augmentations such as Mixup cause a variance shift in the positional embedding of ViT, which has been a hidden factor that degrades the performance of ViT during the test phase. We claim that achieving a stable effect from positional embedding requires a specific condition on the image, which is often broken for the current data augmentation methods. We provide a detailed analysis of this problem as well as the correct configuration for these data augmentations to remove the side effects of variance shift. Experiments showed that adopting our guidelines improves the performance of ViTs compared with the current configuration of data augmentations.