Convexity in ReLU Neural Networks: beyond ICNNs?

TL;DR

This paper characterizes when ReLU neural networks are convex, explores their expressivity compared to ICNNs, and provides a numerical method to verify convexity in complex networks, advancing understanding of convexity constraints in deep learning.

Contribution

It offers necessary and sufficient conditions for ReLU networks to be convex, analyzes their expressivity relative to ICNNs, and introduces a numerical convexity verification method.

Findings

Every convex function by a 1-hidden-layer ReLU network can be represented by an ICNN of the same architecture.

This equivalence does not extend to networks with more than one hidden layer.

A numerical procedure is proposed for exact convexity checking in large ReLU networks.

Abstract

Convex functions and their gradients play a critical role in mathematical imaging, from proximal optimization to Optimal Transport. The successes of deep learning has led many to use learning-based methods, where fixed functions or operators are replaced by learned neural networks. Regardless of their empirical superiority, establishing rigorous guarantees for these methods often requires to impose structural constraints on neural architectures, in particular convexity. The most popular way to do so is to use so-called Input Convex Neural Networks (ICNNs). In order to explore the expressivity of ICNNs, we provide necessary and sufficient conditions for a ReLU neural network to be convex. Such characterizations are based on product of weights and activations, and write nicely for any architecture in the path-lifting framework. As particular applications, we study our characterizations in depth for 1 and 2-hidden-layer neural networks: we show that every convex function implemented by a 1-hidden-layer ReLU network can be also expressed by an ICNN with the same architecture; however this property no longer holds with more layers. Finally, we provide a numerical procedure that allows an exact check of convexity for ReLU neural networks with a large number of affine regions.