Exploring the loss landscape of regularized neural networks via convex duality

TL;DR

This paper analyzes the loss landscape of regularized neural networks by transforming the problem into a convex dual, revealing the structure of solutions, phase transitions, and connectivity properties across different architectures.

Contribution

It introduces a convex duality framework to characterize stationary points, solution sets, and landscape topology of regularized neural networks, including phase transitions and solution nonuniqueness.

Findings

Characterization of stationary points via convex duality.

Identification of phase transitions in global optima topology.

Extension of results to various neural network architectures.

Abstract

We discuss several aspects of the loss landscape of regularized neural networks: the structure of stationary points, connectivity of optimal solutions, path with nonincreasing loss to arbitrary global optimum, and the nonuniqueness of optimal solutions, by casting the problem into an equivalent convex problem and considering its dual. Starting from two-layer neural networks with scalar output, we first characterize the solution set of the convex problem using its dual and further characterize all stationary points. With the characterization, we show that the topology of the global optima goes through a phase transition as the width of the network changes, and construct counterexamples where the problem may have a continuum of optimal solutions. Finally, we show that the solution set characterization and connectivity results can be extended to different architectures, including two-layer vector-valued neural networks and parallel three-layer neural networks.