DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference

TL;DR

DACT-BERT introduces a differentiable adaptive computation time mechanism to BERT, enabling dynamic adjustment of transformer layers during inference for improved efficiency without sacrificing performance.

Contribution

It proposes a novel adaptive computation strategy for BERT that learns to select the optimal number of layers dynamically during inference.

Findings

Reduces computation time while maintaining accuracy.

Outperforms fixed-layer models in efficiency.

Competitive results across various tasks.

Abstract

Large-scale pre-trained language models have shown remarkable results in diverse NLP applications. Unfortunately, these performance gains have been accompanied by a significant increase in computation time and model size, stressing the need to develop new or complementary strategies to increase the efficiency of these models. In this paper we propose DACT-BERT, a differentiable adaptive computation time strategy for BERT-like models. DACT-BERT adds an adaptive computational mechanism to BERT's regular processing pipeline, which controls the number of Transformer blocks that need to be executed at inference time. By doing this, the model learns to combine the most appropriate intermediate representations for the task at hand. Our experiments demonstrate that our approach, when compared to the baselines, excels on a reduced computational regime and is competitive in other less restrictive ones.