Training neural network ensembles via trajectory sampling

TL;DR

This paper introduces a novel method for training neural network ensembles by sampling parameter trajectories biased towards low loss, using techniques from stochastic systems, offering an alternative to gradient-based training.

Contribution

It proposes a new trajectory sampling approach for training neural network ensembles, leveraging stochastic dynamics and biasing techniques, which differs from traditional gradient methods.

Findings

Effective on simple supervised tasks

Potential advantages over gradient-based methods

Demonstrates viability of trajectory sampling approach

Abstract

In machine learning, there is renewed interest in neural network ensembles (NNEs), whereby predictions are obtained as an aggregate from a diverse set of smaller models, rather than from a single larger model. Here, we show how to define and train a NNE using techniques from the study of rare trajectories in stochastic systems. We define an NNE in terms of the trajectory of the model parameters under a simple, and discrete in time, diffusive dynamics, and train the NNE by biasing these trajectories towards a small time-integrated loss, as controlled by appropriate counting fields which act as hyperparameters. We demonstrate the viability of this technique on a range of simple supervised learning tasks. We discuss potential advantages of our trajectory sampling approach compared with more conventional gradient based methods.