Algorithms for solving optimization problems arising from deep neural net models: nonsmooth problems

TL;DR

This paper reviews optimization challenges in deep neural network models, focusing on nonsmooth, nonseparable problems, and presents numerical results on a representative class of such problems.

Contribution

It summarizes key challenges, current methods, and provides numerical insights into nonsmooth, nonseparable optimization problems in deep learning.

Findings

Identifies primary challenges in nonsmooth optimization for neural networks

Reviews current state-of-the-art algorithms and approaches

Provides numerical results demonstrating algorithm performance

Abstract

Machine Learning models incorporating multiple layered learning networks have been seen to provide effective models for various classification problems. The resulting optimization problem to solve for the optimal vector minimizing the empirical risk is, however, highly nonconvex. This alone presents a challenge to application and development of appropriate optimization algorithms for solving the problem. However, in addition, there are a number of interesting problems for which the objective function is non- smooth and nonseparable. In this paper, we summarize the primary challenges involved, the state of the art, and present some numerical results on an interesting and representative class of problems.