Permutative redundancy and uncertainty of the objective in deep learning

TL;DR

This paper discusses how the symmetry and uncertainty in deep learning objectives create numerous equivalent optima, complicating optimization, and explores potential remedies like pruning, reordering, and bio-inspired architectures.

Contribution

It highlights the impact of permutative symmetry and objective uncertainty on optimization landscapes and proposes methods to mitigate ghost optima in deep learning models.

Findings

Traditional architectures have many equivalent global and local optima.

Uncertainty in objectives prevents local optima from being reached.

Proposed remedies can reduce or eliminate ghost optima.

Abstract

Implications of uncertain objective functions and permutative symmetry of traditional deep learning architectures are discussed. It is shown that traditional architectures are polluted by an astronomical number of equivalent global and local optima. Uncertainty of the objective makes local optima unattainable, and, as the size of the network grows, the global optimization landscape likely becomes a tangled web of valleys and ridges. Some remedies which reduce or eliminate ghost optima are discussed including forced pre-pruning, re-ordering, ortho-polynomial activations, and modular bio-inspired architectures.