Early Transformers: A study on Efficient Training of Transformer Models through Early-Bird Lottery Tickets

TL;DR

This paper explores the early-bird ticket hypothesis to efficiently train Transformer models by identifying pruned models early in training, reducing resources needed while maintaining or improving accuracy across various architectures.

Contribution

It introduces a methodology combining iterative pruning and selective retraining to find early-bird tickets in Transformer models, enhancing training efficiency and resource savings.

Findings

Early-bird tickets can be identified within the first few training epochs.

Pruned models from early-bird tickets maintain or surpass original accuracy.

The phenomenon is consistent across different Transformer architectures and tasks.

Abstract

The training of Transformer models has revolutionized natural language processing and computer vision, but it remains a resource-intensive and time-consuming process. This paper investigates the applicability of the early-bird ticket hypothesis to optimize the training efficiency of Transformer models. We propose a methodology that combines iterative pruning, masked distance calculation, and selective retraining to identify early-bird tickets in various Transformer architectures, including ViT, Swin-T, GPT-2, and RoBERTa. Our experimental results demonstrate that early-bird tickets can be consistently found within the first few epochs of training or fine-tuning, enabling significant resource optimization without compromising performance. The pruned models obtained from early-bird tickets achieve comparable or even superior accuracy to their unpruned counterparts while substantially reducing memory usage. Furthermore, our comparative analysis highlights the generalizability of the early-bird ticket phenomenon across different Transformer models and tasks. This research contributes to the development of efficient training strategies for Transformer models, making them more accessible and resource-friendly. By leveraging early-bird tickets, practitioners can accelerate the progress of natural language processing and computer vision applications while reducing the computational burden associated with training Transformer models.