Functional Optimisation of Online Algorithms in Multilayer Neural Networks

TL;DR

This paper explores optimizing online learning algorithms in multilayer neural networks by deriving a modulation function that enhances generalization and reduces training plateaux, with promising simulation results.

Contribution

It introduces a variational approach to determine optimal modulation functions for online learning in multilayer networks, improving upon traditional backpropagation methods.

Findings

Reduced symmetric phase plateaux compared to backpropagation

Derived modulation functions that maximize generalization error decay

Simulation results demonstrate improved learning dynamics

Abstract

We study the online dynamics of learning in fully connected soft committee machines in the student-teacher scenario. The locally optimal modulation function, which determines the learning algorithm, is obtained from a variational argument in such a manner as to maximise the average generalisation error decay per example. Simulations results for the resulting algorithm are presented for a few cases. The symmetric phase plateaux are found to be vastly reduced in comparison to those found when online backpropagation algorithms are used. A discussion of the implementation of these ideas as practical algorithms is given.