SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression

TL;DR

SHRP introduces a structured pruning method for transformer encoders that significantly reduces model size and computation while maintaining high accuracy, enabling more efficient real-time NLP applications.

Contribution

The paper proposes SHRP, a novel framework that automatically prunes redundant attention heads in transformers using a dynamic routing mechanism, improving efficiency without major accuracy loss.

Findings

Achieves 93% of original accuracy with 48% parameter reduction.

Maintains 84% accuracy even when 11/12 layers are pruned.

Delivers 4.2x throughput gain with only 11.5% of original FLOPs.

Abstract

Transformer encoders are widely deployed in large-scale web services for natural language understanding tasks such as text classification, semantic retrieval, and content ranking. However, their high inference latency and memory consumption pose significant challenges for real-time serving and scalability. These limitations stem largely from architectural redundancy, particularly in the attention module. The inherent parameter redundancy of the attention mechanism, coupled with the fact that its attention heads operate with a degree of independence, makes it particularly amenable to structured model compression. In this paper, we propose SHRP (Specialized Head Routing and Pruning), a novel structured pruning framework that automatically identifies and removes redundant attention heads while preserving most of the model's accuracy and compatibility. SHRP introduces Expert Attention, a modular design that treats each attention head as an independent expert, followed by a lightweight shared expander feed-forward network that refines their outputs. The framework employs a unified Top-1 usage-driven mechanism to jointly perform dynamic routing during training and deterministic pruning at deployment. Experimental results on the GLUE benchmark using a BERT-base encoder show that SHRP achieves 93% of the original model accuracy while reducing parameters by 48 percent. Under an extreme compression scenario where 11/12 of the layers are pruned, the model still maintains 84% accuracy and delivers a 4.2x throughput gain while reducing computation to as low as 11.5 percent of the original FLOPs, demonstrating its practical utility for large-scale and latency-sensitive web deployments.