Explicit Multi-head Attention for Inter-head Interaction in Large Language Models

TL;DR

This paper introduces Multi-head Explicit Attention (MEA), a novel attention variant that explicitly models cross-head interactions, improving robustness, efficiency, and performance in large language models.

Contribution

The paper proposes MEA with HLC and normalization layers, enabling rich inter-head communication, faster convergence, and efficient KV-cache compression in large language models.

Findings

MEA improves attention performance and robustness.

Faster convergence with larger learning rates.

50% reduction in KV-cache memory with minimal performance loss.

Abstract

In large language models built upon the Transformer architecture, recent studies have shown that inter-head interaction can enhance attention performance. Motivated by this, we propose Multi-head Explicit Attention (MEA), a simple yet effective attention variant that explicitly models cross-head interaction. MEA consists of two key components: a Head-level Linear Composition (HLC) module that separately applies learnable linear combinations to the key and value vectors across heads, thereby enabling rich inter-head communication; and a head-level Group Normalization layer that aligns the statistical properties of the recombined heads. MEA shows strong robustness in pretraining, which allows the use of larger learning rates that lead to faster convergence, ultimately resulting in lower validation loss and improved performance across a range of tasks. Furthermore, we explore the parameter efficiency of MEA by reducing the number of attention heads and leveraging HLC to reconstruct them using low-rank "virtual heads". This enables a practical key-value cache compression strategy that reduces KV-cache memory usage by 50% with negligible performance loss on knowledge-intensive and scientific reasoning tasks, and only a 3.59% accuracy drop for Olympiad-level mathematical benchmarks.