LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

TL;DR

LayerBoost introduces a layer-aware attention reduction method that selectively modifies transformer layers to improve inference efficiency while maintaining model performance.

Contribution

It systematically analyzes layer sensitivities to apply tailored attention modifications, combining softmax, linear, or no attention per layer, with a lightweight distillation recovery.

Findings

Reduces inference latency and improves throughput by up to 68%.

Maintains competitive performance on several benchmarks.

Outperforms state-of-the-art attention linearization methods.

Abstract

Transformers are mostly relying on softmax attention, which introduces quadratic complexity with respect to sequence length and remains a major bottleneck for efficient inference. Prior work on linear or hybrid attention typically replaces softmax attention uniformly across all layers, often leading to significant performance degradation or requiring extensive retraining to recover model quality. This work proposes LayerBoost, a layer-aware attention reduction method that selectively modifies the attention mechanism based on the sensitivity of individual transformer layers. It first performs a systematic sensitivity analysis on a pretrained model to identify layers that are critical for maintaining performance. Guided by this analysis, three distinct strategies can be applied: retaining standard softmax attention in highly sensitive layers, replacing it with linear sliding window attention in moderately sensitive layers, and removing attention entirely in layers that exhibit low sensitivity. To recover performance after these architectural modifications, we introduce a lightweight distillation-based healing phase requiring only 10M additional training tokens. LayerBoost reduces inference latency and improves throughput by up to 68% at high concurrency, while maintaining competitive model quality. It matches base model performance on several benchmarks, exhibits only minor degradations on others, and significantly outperforms state-of-the-art attention linearization methods. These efficiency gains make our method particularly well-suited for high-concurrency serving and hardware-constrained deployment scenarios, where inference cost and memory footprint are critical bottlenecks.