Parallel Deep Neural Networks Have Zero Duality Gap

TL;DR

This paper proves that deep neural networks with parallel architectures and specific regularizations can achieve zero duality gap, enabling convex reformulations and globally optimal training, unlike traditional deep networks.

Contribution

It introduces a parallel architecture approach with modified regularization to attain strong duality in deep networks, extending beyond shallow cases.

Findings

Zero duality gap for parallel deep networks with regularization.

Convex reformulation enables globally optimal training.

Regularization encourages low-rank solutions and strong duality in certain cases.

Abstract

Training deep neural networks is a challenging non-convex optimization problem. Recent work has proven that the strong duality holds (which means zero duality gap) for regularized finite-width two-layer ReLU networks and consequently provided an equivalent convex training problem. However, extending this result to deeper networks remains to be an open problem. In this paper, we prove that the duality gap for deeper linear networks with vector outputs is non-zero. In contrast, we show that the zero duality gap can be obtained by stacking standard deep networks in parallel, which we call a parallel architecture, and modifying the regularization. Therefore, we prove the strong duality and existence of equivalent convex problems that enable globally optimal training of deep networks. As a by-product of our analysis, we demonstrate that the weight decay regularization on the network parameters explicitly encourages low-rank solutions via closed-form expressions. In addition, we show that strong duality holds for three-layer standard ReLU networks given rank-1 data matrices.