KKT-based optimality conditions for neural network approximation

TL;DR

This paper derives necessary optimality conditions for neural network approximation in $l_1$ and max norms, using KKT conditions and convex analysis, focusing on shallow networks with one hidden layer.

Contribution

It introduces a novel approach to formulate nonsmooth neural network approximation problems as smooth constrained problems and applies KKT conditions for optimality analysis.

Findings

Provides necessary optimality conditions for neural network approximation.

Reformulates nonsmooth problems into smooth constrained optimization problems.

Uses convex analysis to express optimality conditions.

Abstract

In this paper, we obtain necessary optimality conditions for neural network approximation. We consider neural networks in Manhattan ($l_1$ norm) and Chebyshev ($\max$ norm). The optimality conditions are based on neural networks with at most one hidden layer. We reformulate nonsmooth unconstrained optimisation problems as larger dimension constrained problems with smooth objective functions and constraints. Then we use KKT conditions to develop the necessary conditions and present the optimality conditions in terms of convex analysis and convex sets.