A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization

TL;DR

This paper demonstrates how variable elimination reshapes the optimization landscape in non-convex problems, transforming saddle points into local maxima, thereby improving convergence and stability in machine learning models.

Contribution

It provides a rigorous geometric analysis of variable elimination, showing how it simplifies the landscape by converting saddle points into local maxima, and validates this in neural network training.

Findings

Variable elimination transforms saddle points into local maxima.

The approach improves stability and convergence in deep residual networks.

Landscape simplification guides the design of robust optimization algorithms.

Abstract

The proliferation of saddle points, rather than poor local minima, is increasingly understood to be a primary obstacle in large-scale non-convex optimization for machine learning. Variable elimination algorithms, like Variable Projection (VarPro), have long been observed to exhibit superior convergence and robustness in practice, yet a principled understanding of why they so effectively navigate these complex energy landscapes has remained elusive. In this work, we provide a rigorous geometric explanation by comparing the optimization landscapes of the original and reduced formulations. Through a rigorous analysis based on Hessian inertia and the Schur complement, we prove that variable elimination fundamentally reshapes the critical point structure of the objective function, revealing that local maxima in the reduced landscape are created from, and correspond directly to, saddle points in the original formulation. Our findings are illustrated on the canonical problem of non-convex matrix factorization, visualized directly on two-parameter neural networks, and finally validated in training deep Residual Networks, where our approach yields dramatic improvements in stability and convergence to superior minima. This work goes beyond explaining an existing method; it establishes landscape simplification via saddle point transformation as a powerful principle that can guide the design of a new generation of more robust and efficient optimization algorithms.