Kalman Bayesian Neural Networks for Closed-form Online Learning

TL;DR

This paper introduces a novel closed-form Bayesian inference method for neural networks that enables efficient online learning by treating weight updates as Bayesian filtering, avoiding gradient descent.

Contribution

The paper presents a new approach for Bayesian neural network training using closed-form solutions, simplifying online learning without gradient-based optimization.

Findings

Effective on UCI datasets with competitive performance

Enables sequential and online training of BNNs

Reduces computational complexity compared to traditional methods

Abstract

Compared to point estimates calculated by standard neural networks, Bayesian neural networks (BNN) provide probability distributions over the output predictions and model parameters, i.e., the weights. Training the weight distribution of a BNN, however, is more involved due to the intractability of the underlying Bayesian inference problem and thus, requires efficient approximations. In this paper, we propose a novel approach for BNN learning via closed-form Bayesian inference. For this purpose, the calculation of the predictive distribution of the output and the update of the weight distribution are treated as Bayesian filtering and smoothing problems, where the weights are modeled as Gaussian random variables. This allows closed-form expressions for training the network's parameters in a sequential/online fashion without gradient descent. We demonstrate our method on several UCI datasets and compare it to the state of the art.