Query-Key Normalization for Transformers

TL;DR

This paper introduces QKNorm, a novel normalization method for Transformers that improves low-resource language translation by stabilizing attention mechanisms, leading to significant BLEU score improvements.

Contribution

QKNorm modifies the attention mechanism with $ ext{l}_2$ normalization and learnable scaling, enhancing translation quality in low-resource settings.

Findings

Average BLEU improvement of 0.928 over state-of-the-art models

Effective normalization technique for low-resource translation tasks

Improves attention stability without reducing model expressivity

Abstract

Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNorm, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply $\ell_2$ normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15.