DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models

TL;DR

This paper introduces DEED, a method for reducing inference latency in encoder-decoder transformers for vision-language tasks by enabling dynamic early exits at different decoder layers during decoding.

Contribution

The paper proposes a multi-exit transformer model trained with deep supervision and a step-level dynamic early exit strategy to accelerate inference without sacrificing accuracy.

Findings

Inference latency reduced by 30%-60%.

Maintains comparable or higher accuracy with early exits.

Effective on multiple vision-language tasks.

Abstract

Encoder-decoder transformer models have achieved great success on various vision-language (VL) tasks, but they suffer from high inference latency. Typically, the decoder takes up most of the latency because of the auto-regressive decoding. To accelerate the inference, we propose an approach of performing Dynamic Early Exit on Decoder (DEED). We build a multi-exit encoder-decoder transformer model which is trained with deep supervision so that each of its decoder layers is capable of generating plausible predictions. In addition, we leverage simple yet practical techniques, including shared generation head and adaptation modules, to keep accuracy when exiting at shallow decoder layers. Based on the multi-exit model, we perform step-level dynamic early exit during inference, where the model may decide to use fewer decoder layers based on its confidence of the current layer at each individual decoding step. Considering different number of decoder layers may be used at different decoding steps, we compute deeper-layer decoder features of previous decoding steps just-in-time, which ensures the features from different decoding steps are semantically aligned. We evaluate our approach with two state-of-the-art encoder-decoder transformer models on various VL tasks. We show our approach can reduce overall inference latency by 30%-60% with comparable or even higher accuracy compared to baselines.