A Bayesian Approach to Recurrence in Neural Networks

TL;DR

This paper introduces a Bayesian recurrent unit derived from Bayes's theorem, enabling inference that considers both past and future inputs, and demonstrates competitive or superior performance in speech recognition tasks.

Contribution

It presents a novel Bayesian recurrent architecture with a feedback mechanism inspired by Bayesian inference, unifying forward and backward processing in neural networks.

Findings

Performs as well as bidirectional RNNs with the same number of parameters.

Can surpass traditional bidirectional recurrence when explicitly configured bidirectionally.

Uses a probabilistic approach to incorporate future context in recurrent neural networks.

Abstract

We begin by reiterating that common neural network activation functions have simple Bayesian origins. In this spirit, we go on to show that Bayes's theorem also implies a simple recurrence relation; this leads to a Bayesian recurrent unit with a prescribed feedback formulation. We show that introduction of a context indicator leads to a variable feedback that is similar to the forget mechanism in conventional recurrent units. A similar approach leads to a probabilistic input gate. The Bayesian formulation leads naturally to the two pass algorithm of the Kalman smoother or forward-backward algorithm, meaning that inference naturally depends upon future inputs as well as past ones. Experiments on speech recognition confirm that the resulting architecture can perform as well as a bidirectional recurrent network with the same number of parameters as a unidirectional one. Further, when configured explicitly bidirectionally, the architecture can exceed the performance of a conventional bidirectional recurrence.