On Compression Principle and Bayesian Optimization for Neural Networks

TL;DR

This paper introduces a compression-based principle for neural network modeling, utilizing Bayesian methods and variational approximations to optimize model complexity and improve generalization.

Contribution

It proposes a novel compression principle for predictive models and develops Bayesian Stochastic Gradient Descent for hyper-parameter optimization.

Findings

Dropout enables continuous dimensionality reduction.

BSGD effectively optimizes hyper-parameters with minimal settings.

The approach improves model generalization through compression-based criteria.

Abstract

Finding methods for making generalizable predictions is a fundamental problem of machine learning. By looking into similarities between the prediction problem for unknown data and the lossless compression we have found an approach that gives a solution. In this paper we propose a compression principle that states that an optimal predictive model is the one that minimizes a total compressed message length of all data and model definition while guarantees decodability. Following the compression principle we use Bayesian approach to build probabilistic models of data and network definitions. A method to approximate Bayesian integrals using a sequence of variational approximations is implemented as an optimizer for hyper-parameters: Bayesian Stochastic Gradient Descent (BSGD). Training with BSGD is completely defined by setting only three parameters: number of epochs, the size of the dataset and the size of the minibatch, which define a learning rate and a number of iterations. We show that dropout can be used for a continuous dimensionality reduction that allows to find optimal network dimensions as required by the compression principle.