Tricks and Plug-ins for Gradient Boosting with Transformers

TL;DR

This paper introduces BoostTransformer, a framework that enhances transformer models with boosting techniques like subgrid token selection and importance-weighted sampling, leading to faster training and better accuracy in NLP tasks.

Contribution

The paper presents a novel boosting-based framework for transformers that improves efficiency and performance without extensive hyperparameter tuning.

Findings

Faster convergence compared to standard transformers

Higher accuracy on fine-grained text classification benchmarks

Reduced architectural search overhead

Abstract

Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead.