Bayesian Recurrent Units and the Forward-Backward Algorithm

TL;DR

This paper introduces Bayesian recurrent units derived from Bayes's theorem, which can be integrated into neural networks to enhance speech recognition performance with minimal additional parameters.

Contribution

It presents a novel theoretical framework linking Bayesian recurrence with neural networks, inspired by hidden Markov models, and demonstrates practical benefits in speech recognition.

Findings

Improved speech recognition accuracy with Bayesian units

Low additional computational cost

Theoretical connection between Bayesian recurrence and HMMs

Abstract

Using Bayes's theorem, we derive a unit-wise recurrence as well as a backward recursion similar to the forward-backward algorithm. The resulting Bayesian recurrent units can be integrated as recurrent neural networks within deep learning frameworks, while retaining a probabilistic interpretation from the direct correspondence with hidden Markov models. Whilst the contribution is mainly theoretical, experiments on speech recognition indicate that adding the derived units at the end of state-of-the-art recurrent architectures can improve the performance at a very low cost in terms of trainable parameters.