Estimating the Number of Components in a Mixture of Multilayer Perceptrons

TL;DR

This paper extends the BIC criterion to estimate the number of components in mixtures of multilayer perceptrons and proves its convergence, supported by theoretical analysis and numerical examples.

Contribution

It introduces a penalized-likelihood criterion for multilayer perceptron mixtures and establishes its convergence properties.

Findings

Proves convergence of the BIC criterion in multilayer perceptron mixtures.

Extends penalized-likelihood methods to neural network models.

Provides numerical illustrations of the theoretical results.

Abstract

BIC criterion is widely used by the neural-network community for model selection tasks, although its convergence properties are not always theoretically established. In this paper we will focus on estimating the number of components in a mixture of multilayer perceptrons and proving the convergence of the BIC criterion in this frame. The penalized marginal-likelihood for mixture models and hidden Markov models introduced by Keribin (2000) and, respectively, Gassiat (2002) is extended to mixtures of multilayer perceptrons for which a penalized-likelihood criterion is proposed. We prove its convergence under some hypothesis which involve essentially the bracketing entropy of the generalized score-functions class and illustrate it by some numerical examples.