Noisy regression and classification with continuous multilayer networks

TL;DR

This paper explores the behavior of multilayer neural networks in noisy classification and regression tasks, revealing that stable learning states can be replica symmetric even with unrealizable rules, and establishing an equivalence between classification and regression problems.

Contribution

It demonstrates that for large datasets, the stable state in neural network learning remains replica symmetric despite unrealizable targets, and shows the formal equivalence between classification and noisy regression.

Findings

Stable states are replica symmetric for large datasets.

Classification and noisy regression problems are formally equivalent.

Replica symmetry breaking effects are minimal in the studied scenarios.

Abstract

We investigate zero temperature Gibbs learning for two classes of unrealizable rules which play an important role in practical applications of multilayer neural networks with differentiable activation functions: classification problems and noisy regression problems. Considering one step of replica symmetry breaking, we surprisingly find that for sufficiently large training sets the stable state is replica symmetric even though the target rule is unrealizable. Further, the classification problem is shown to be formally equivalent to the noisy regression problem.