Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networks

TL;DR

This paper reviews recent smoothness conditions for non-convex optimization in deep learning, categorizes them, and assesses their relevance to training deep linear neural networks for binary classification.

Contribution

It systematically orders and evaluates various non-convex smoothness conditions and their applicability to deep linear neural network training.

Findings

Different smoothness conditions are applicable to deep linear networks.

The paper provides criteria to determine the validity of these conditions.

Applicability varies depending on the specific smoothness assumption.

Abstract

The presence of non-convexity in smooth optimization problems arising from deep learning have sparked new smoothness conditions in the literature and corresponding convergence analyses. We discuss these smoothness conditions, order them, provide conditions for determining whether they hold, and evaluate their applicability to training a deep linear neural network for binary classification.