Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models

TL;DR

This paper explores layer dropping in large language models during inference, showing that removing certain layers can significantly improve speed with minimal performance loss, especially when dropping attention layers.

Contribution

It introduces a method of dropping specific layers, particularly attention layers, during inference to reduce latency with minimal accuracy degradation.

Findings

Dropping 33% of attention layers causes only 1.8% performance drop.

Removing entire layers yields significant speedups with marginal performance impact.

Skipping layers except the latter ones affects performance more as more layers are skipped.

Abstract

The inference demand for LLMs has skyrocketed in recent months, and serving models with low latencies remains challenging due to the quadratic input length complexity of the attention layers. In this work, we investigate the effect of dropping MLP and attention layers at inference time on the performance of Llama-v2 models. We find that dropping dreeper attention layers only marginally decreases performance but leads to the best speedups alongside dropping entire layers. For example, removing 33\% of attention layers in a 13B Llama2 model results in a 1.8\% drop in average performance over the OpenLLM benchmark. We also observe that skipping layers except the latter layers reduces performances for more layers skipped, except for skipping the attention layers.