Variable reduction as a nonlinear preconditioning approach for optimization problems

TL;DR

This paper introduces a nonlinear variable reduction method that improves optimization efficiency by constraining variables to satisfy certain optimality conditions, leading to better-conditioned problems and enhanced gradient descent performance.

Contribution

The paper proposes a novel nonlinear preconditioning approach that reduces variables via optimality constraints, improving problem conditioning and optimization efficiency.

Findings

The reduced objective function has better conditioning.

Preconditioned gradient descent shows improved convergence.

Numerical results support the theoretical benefits.

Abstract

When considering an unconstrained minimization problem, a standard approach is to solve the optimality system with a Newton method possibly preconditioned by, e.g., nonlinear elimination. In this contribution, we argue that nonlinear elimination could be used to reduce the number of optimization variables by artificially constraining them to satisfy a subset of the optimality conditions. Consequently, a reduced objective function is derived which can now be minimized with any optimization algorithm. By choosing suitable variables to eliminate, the conditioning of the reduced optimization problem is largely improved. We here focus in particular on a right preconditioned gradient descent and show theoretical and numerical results supporting the validity of the presented approach.