Holonorm

TL;DR

Holonorm is a novel normalization method for transformers that preserves orthogonality and stability, addressing limitations of Tanh-based normalization by mapping vectors into the open unit ball.

Contribution

The paper introduces Holonorm, a new normalization technique with residual connections and nonlinearity, suitable for replacing Tanh in transformer models, improving stability and interpretability.

Findings

Holonorm preserves orthogonality and invertibility.

Holonorm maps vectors into the open unit ball, preventing exploding activations.

Holonorm enhances stability in deep transformer models.

Abstract

Normalization is a key point in transformer training . In Dynamic Tanh (DyT), the author demonstrated that Tanh can be used as an alternative layer normalization (LN) and confirmed the effectiveness of the idea. But Tanh itself faces orthogonality, linearity and distortion problems. Due to that, his proposition cannot be reliable. So we propose a Holonorm (hn) which has residual connections and nonlinearity. Holonorm is suitable for replacing Tanh in the context of normalization. Although the HoloNorm expression could be similar to the softsign function in dimension one, softsign is a componentwise function which is not good for tensors and vectors of great dimension. Holonorm preserves the orthogonality, the direction, the invertibility of the signal. Holonorm is also a suitable metric, maps all vectors into the open unit ball. This prevents exploding activations and improves stability in deep Transformer models. In this work, we have meticulously examined the normalization in transformers and say that Holonorm, a generalized form of softsign function suited as a normalization function first.Second, defined between 0 and 1 hn serves as a percentage, and $1 - \text{Holonorm}$ is its complement, making it better understandable in evaluating a model.