Training Noise Token Pruning

TL;DR

This paper introduces Training Noise Token (TNT) Pruning, a method for efficiently reducing tokens in vision transformers by using continuous noise during training, leading to better optimization and deployment efficiency.

Contribution

TNT Pruning is a novel approach that relaxes discrete token dropping with continuous noise, improving training and deployment in vision transformers.

Findings

TNT outperforms previous pruning methods on ImageNet.

Theoretical links to Rate-Distortion theory are established.

Empirical results show improved efficiency and accuracy.

Abstract

In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods.