Modelling the Probability Density of Markov Sources

TL;DR

This paper proposes an objective function for efficiently encoding Markov sources, which can optimize neural network recognition and generative models, unifying previous multilayer network optimization approaches.

Contribution

It introduces a new objective function for encoding Markov sources that unifies and extends existing neural network optimization methods.

Findings

The objective minimizes the average bits needed for joint state encoding.

It provides a framework for optimizing recognition and generative models in multilayer networks.

Unifies previous results on multilayer neural network optimization.

Abstract

This paper introduces an objective function that seeks to minimise the average total number of bits required to encode the joint state of all of the layers of a Markov source. This type of encoder may be applied to the problem of optimising the bottom-up (recognition model) and top-down (generative model) connections in a multilayer neural network, and it unifies several previous results on the optimisation of multilayer neural networks.