Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations

TL;DR

This paper extends the theoretical understanding of monotonic neural networks, demonstrating their universal approximation capabilities with new activation and weight constraints, and proposes an improved training method for better optimization.

Contribution

It generalizes previous results to broader classes of monotonic networks, introduces a new activation adjustment approach, and simplifies training while maintaining universality.

Findings

Monotonic MLPs with convex activations are universal approximators.

The proposed activation adjustment improves training stability.

Experimental results favor the new approach over traditional methods.

Abstract

Conventional techniques for imposing monotonicity in MLPs by construction involve the use of non-negative weight constraints and bounded activation functions, which pose well-known optimization challenges. In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions. Additionally, we show an equivalence between the saturation side in the activations and the sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts. Our results provide theoretical grounding to the empirical effectiveness observed in previous works while leading to possible architectural simplification. Moreover, to further alleviate the optimization difficulties, we propose an alternative formulation that allows the network to adjust its activations according to the sign of the weights. This eliminates the requirement for weight reparameterization, easing initialization and improving training stability. Experimental evaluation reinforces the validity of the theoretical results, showing that our novel approach compares favourably to traditional monotonic architectures.