Smooth Min-Max Monotonic Networks

TL;DR

This paper introduces a smooth min-max (SMM) neural network module that enforces monotonicity, improves training stability, and maintains competitive generalization performance, offering a simple and efficient alternative for monotonic modeling.

Contribution

The authors propose a novel smooth min-max (SMM) network module using strictly-increasing smooth functions, addressing training issues of traditional min-max networks while preserving their approximation capabilities.

Findings

SMM networks alleviate local optima problems in training.

SMM maintains approximation properties of traditional MM networks.

SMM achieves comparable generalization performance to existing methods.

Abstract

Monotonicity constraints are powerful regularizers in statistical modelling. They can support fairness in computer-aided decision making and increase plausibility in data-driven scientific models. The seminal min-max (MM) neural network architecture ensures monotonicity, but often gets stuck in undesired local optima during training because of partial derivatives of the MM nonlinearities being zero. We propose a simple modification of the MM network using strictly-increasing smooth minimum and maximum functions that alleviates this problem. The resulting smooth min-max (SMM) network module inherits the asymptotic approximation properties from the MM architecture. It can be used within larger deep learning systems trained end-to-end. The SMM module is conceptually simple and computationally less demanding than state-of-the-art neural networks for monotonic modelling. Our experiments show that this does not come with a loss in generalization performance compared to alternative neural and non-neural approaches.