Reversible Jump MCMC Simulated Annealing for Neural Networks

TL;DR

This paper introduces a reversible jump MCMC simulated annealing algorithm for optimizing RBF networks, enabling global search over network parameters and size, with theoretical and empirical convergence guarantees.

Contribution

It presents a novel algorithm combining reversible jump MCMC and simulated annealing for neural network optimization, integrating model selection criteria within a Bayesian framework.

Findings

Algorithm converges efficiently to posterior modes.

Supports model selection criteria like AIC, BIC, MDL.

Demonstrates effectiveness through empirical tests.

Abstract

We propose a novel reversible jump Markov chain Monte Carlo (MCMC) simulated annealing algorithm to optimize radial basis function (RBF) networks. This algorithm enables us to maximize the joint posterior distribution of the network parameters and the number of basis functions. It performs a global search in the joint space of the parameters and number of parameters, thereby surmounting the problem of local minima. We also show that by calibrating a Bayesian model, we can obtain the classical AIC, BIC and MDL model selection criteria within a penalized likelihood framework. Finally, we show theoretically and empirically that the algorithm converges to the modes of the full posterior distribution in an efficient way.