Tightening convex relaxations of trained neural networks: a unified approach for convex and S-shaped activations

TL;DR

This paper introduces a recursive method to produce tight convex relaxations for neural networks with convex or S-shaped activations, enhancing optimization integration.

Contribution

It develops a unified recursive formula for convexifying a broad class of activation functions, improving upon previous convex relaxation techniques.

Findings

Provides a recursive formula for convexification of S-shaped activations.

Enables efficient computation of separating hyperplanes in neural network relaxations.

Extends convex relaxation methods to non-polyhedral activation functions.

Abstract

The non-convex nature of trained neural networks has created significant obstacles in their incorporation into optimization models. In this context, Anderson et al. (2020) provided a framework to obtain the convex hull of the graph of a piecewise linear convex activation function composed with an affine function; this effectively convexifies activations such as the ReLU together with the affine transformation that precedes it. In this article, we contribute to this line of work by developing a recursive formula that yields a tight convexification for the composition of an activation with an affine function for a wide scope of activation functions, namely, convex or ``S-shaped". Our approach can be used to efficiently compute separating hyperplanes or determine that none exists in various settings, including non-polyhedral cases.