The Curious Case of Convex Neural Networks

TL;DR

This paper explores convex neural networks with convexity constraints, demonstrating their regularization benefits, competitive performance, and robustness to noisy labels across image classification tasks.

Contribution

It introduces a method to enforce convexity in neural networks, improving generalization and robustness while maintaining competitive accuracy.

Findings

Convex neural networks self-regularize and reduce overfitting.

They outperform unconstrained MLPs and match CNN performance.

They are robust to label noise in training data.

Abstract

In this paper, we investigate a constrained formulation of neural networks where the output is a convex function of the input. We show that the convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The convexity constraints include restricting the weights (for all but the first layer) to be non-negative and using a non-decreasing convex activation function. Albeit simple, these constraints have profound implications on the generalization abilities of the network. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures and (c) IOC-NNs show robustness to noise in train labels. We demonstrate the efficacy of the proposed idea using thorough experiments and ablation studies on standard image classification datasets with three different neural network architectures.