DPBERT: Efficient Inference for BERT based on Dynamic Planning

TL;DR

DPBERT introduces a dynamic planning strategy that selectively skips transformer layers during inference, significantly reducing latency while preserving high accuracy, thus enabling more efficient BERT deployment on resource-constrained devices.

Contribution

It presents a novel fine-tuning approach with a planning module that adaptively skips layers, improving inference efficiency without sacrificing accuracy.

Findings

Reduces BERT inference latency to 75% of original

Maintains 98% of original BERT accuracy

Outperforms existing input-adaptive methods in speed-accuracy trade-off

Abstract

Large-scale pre-trained language models such as BERT have contributed significantly to the development of NLP. However, those models require large computational resources, making it difficult to be applied to mobile devices where computing power is limited. In this paper we aim to address the weakness of existing input-adaptive inference methods which fail to take full advantage of the structure of BERT. We propose Dynamic Planning in BERT, a novel fine-tuning strategy that can accelerate the inference process of BERT through selecting a subsequence of transformer layers list of backbone as a computational path for an input sample. To do this, our approach adds a planning module to the original BERT model to determine whether a layer is included or bypassed during inference. Experimental results on the GLUE benchmark exhibit that our method reduces latency to 75\% while maintaining 98\% accuracy, yielding a better accuracy-speed trade-off compared to state-of-the-art input-adaptive methods.