Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Network

TL;DR

This paper introduces an algebraic method to analyze the landscape of ridge-regularized MSE in ReLU perceptrons, revealing all local minima including higher-dimensional sets, through systematic enumeration.

Contribution

It develops a novel algebraic framework and enumeration strategy to identify all local minima of RR-MSE in minimal ReLU perceptrons, including higher-dimensional solutions.

Findings

Identifies all local minima, including higher-dimensional sets.

Demonstrates the approach on minimal perceptrons with few hidden units.

Shows the algebraic method's potential despite computational intensity.

Abstract

This paper investigates a perceptron, a simple neural network model, with ReLU activation and a ridge-regularized mean squared error (RR-MSE). Our approach leverages the fact that the RR-MSE for ReLU perceptron is piecewise polynomial, enabling a systematic analysis using tools from computational algebra. In particular, we develop a Divide-Enumerate-Merge strategy that exhaustively enumerates all local minima of the RR-MSE. By virtue of the algebraic formulation, our approach can identify not only the typical zero-dimensional minima (i.e., isolated points) obtained by numerical optimization, but also higher-dimensional minima (i.e., connected sets such as curves, surfaces, or hypersurfaces). Although computational algebraic methods are computationally very intensive for perceptrons of practical size, as a proof of concept, we apply the proposed approach in practice to minimal perceptrons with a few hidden units.