A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness

TL;DR

This paper introduces a novel framework for controlling neural network sensitivity through direct parameterization of bi-Lipschitzness, enabling better theoretical understanding and practical applications such as uncertainty estimation.

Contribution

It proposes a new bi-Lipschitz control method using convex neural networks and Legendre-Fenchel duality, providing precise, simple, and theoretically grounded control of network sensitivity.

Findings

Demonstrates effective control of neural network sensitivity.

Shows broad applicability in uncertainty estimation.

Provides theoretical analysis supporting the framework.

Abstract

While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of \emph{bi-Lipschitzness} has been proved as a beneficial inductive bias in many areas. However, due to its complexity, the design and control of bi-Lipschitz architectures are falling behind, and a model that is precisely designed for bi-Lipschitzness realizing a direct and simple control of the constants along with solid theoretical analysis is lacking. In this work, we investigate and propose a novel framework for bi-Lipschitzness that can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality. Its desirable properties are illustrated with concrete experiments. We also apply this framework to uncertainty estimation and monotone problem settings to illustrate its broad range of applications.