$E(2)$-Equivariant Vision Transformer

TL;DR

This paper introduces GE-ViT, a vision transformer that incorporates a novel positional encoding to achieve equivariance, backed by theoretical proof and superior experimental performance on benchmark datasets.

Contribution

The paper proposes a new group equivariant vision transformer with a novel positional encoding operator that ensures theoretical equivariance.

Findings

GE-ViT outperforms non-equivariant models on benchmarks

Theoretical proof confirms equivariance of GE-ViT

Effective positional encoding enhances learning of data symmetries

Abstract

Vision Transformer (ViT) has achieved remarkable performance in computer vision. However, positional encoding in ViT makes it substantially difficult to learn the intrinsic equivariance in data. Initial attempts have been made on designing equivariant ViT but are proved defective in some cases in this paper. To address this issue, we design a Group Equivariant Vision Transformer (GE-ViT) via a novel, effective positional encoding operator. We prove that GE-ViT meets all the theoretical requirements of an equivariant neural network. Comprehensive experiments are conducted on standard benchmark datasets, demonstrating that GE-ViT significantly outperforms non-equivariant self-attention networks. The code is available at https://github.com/ZJUCDSYangKaifan/GEVit.