Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

TL;DR

This paper introduces Streaming Normalization, a unified, simple, and biologically-plausible normalization method applicable across various learning scenarios, improving upon Batch Normalization especially for online and recurrent learning.

Contribution

The authors propose a generalized normalization framework that addresses limitations of Batch Normalization, including online and recurrent learning, and introduce Lp Normalization, notably L1, for efficiency and biological plausibility.

Findings

Streaming Normalization outperforms existing methods in diverse scenarios.

L1 Normalization is effective, simple, and hardware-friendly.

The approach is applicable to all learning paradigms.

Abstract

We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed --- recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations.