Deep Neural Network Structures Solving Variational Inequalities

TL;DR

This paper explores the connection between deep neural network structures and variational inequalities, showing that many activation functions are proximity operators and analyzing the convergence of related models.

Contribution

It introduces a novel framework linking neural network activations to proximity operators and studies the asymptotic behavior of these composite models.

Findings

Activation functions in neural networks are proximity operators.

Conditions for averagedness of composite models are established.

Limit processes solve variational inequalities not necessarily from minimization.

Abstract

Motivated by structures that appear in deep neural networks, we investigate nonlinear composite models alternating proximity and affine operators defined on different spaces. We first show that a wide range of activation operators used in neural networks are actually proximity operators. We then establish conditions for the averagedness of the proposed composite constructs and investigate their asymptotic properties. It is shown that the limit of the resulting process solves a variational inequality which, in general, does not derive from a minimization problem.