Enhanced QKNorm normalization for neural transformers with the Lp norm

TL;DR

This paper introduces a generalized normalization method for neural transformers using the Lp norm, aiming to improve stability in learning by extending existing QKNorm normalization to non-Euclidean norms.

Contribution

It proposes a novel generalization of QKNorm normalization based on the Lp norm, broadening the scope of normalization techniques in transformer models.

Findings

Demonstrates the method's suitability on a simple problem

Shows potential for improved stability in transformer training

Extends normalization techniques beyond Euclidean norms

Abstract

The normalization of query and key vectors is an essential part of the Transformer architecture. It ensures that learning is stable regardless of the scale of these vectors. Some normalization approaches are available. In this preliminary work, a generalization of the QKNorm normalization scheme is proposed. The approach is based on the Lp norm, allowing non-Euclidean norms to be employed. Experimental results demonstrate the suitability of the method for a simple problem.