Gaussian Gated Linear Networks

TL;DR

The paper introduces Gaussian Gated Linear Networks (G-GLN), a novel neural network model that learns efficiently without backpropagation, offering universality, interpretability, and robustness, with strong performance on regression and density estimation tasks.

Contribution

It extends the GLN framework to regression and density modeling using Gaussian mixing, demonstrating competitive performance and practical applicability.

Findings

Achieves state-of-the-art results on regression benchmarks.

Effective in online learning and density estimation tasks.

Robust and interpretable model with local credit assignment.

Abstract

We propose the Gaussian Gated Linear Network (G-GLN), an extension to the recently proposed GLN family of deep neural networks. Instead of using backpropagation to learn features, GLNs have a distributed and local credit assignment mechanism based on optimizing a convex objective. This gives rise to many desirable properties including universality, data-efficient online learning, trivial interpretability and robustness to catastrophic forgetting. We extend the GLN framework from classification to multiple regression and density modelling by generalizing geometric mixing to a product of Gaussian densities. The G-GLN achieves competitive or state-of-the-art performance on several univariate and multivariate regression benchmarks, and we demonstrate its applicability to practical tasks including online contextual bandits and density estimation via denoising.