Inexact Newton-type Methods for Optimisation with Nonnegativity Constraints

TL;DR

This paper develops inexact Newton-type methods within a two-metric projection framework to efficiently solve large-scale nonconvex optimization problems with nonnegativity constraints, outperforming traditional methods in convergence and practical performance.

Contribution

It extends the two-metric projection framework with inexact subproblem solutions and non-positive curvature directions for large-scale nonconvex problems, achieving state-of-the-art convergence rates.

Findings

Achieves improved convergence rates for nonconvex problems.

Demonstrates competitive practical performance on various applications.

Effectively handles large-scale nonnegativity-constrained optimization tasks.

Abstract

We consider solving large scale nonconvex optimisation problems with nonnegativity constraints. Such problems arise frequently in machine learning, such as nonnegative least-squares, nonnegative matrix factorisation, as well as problems with sparsity-inducing regularisation. In such settings, first-order methods, despite their simplicity, can be prohibitively slow on ill-conditioned problems or become trapped near saddle regions, while most second-order alternatives involve non-trivially challenging subproblems. The two-metric projection framework, initially proposed by Bertsekas (1982), alleviates these issues and achieves the best of both worlds by combining projected gradient steps at the boundary of the feasible region with Newton steps in the interior in such a way that feasibility can be maintained by simple projection onto the nonnegative orthant. We develop extensions of the two-metric projection framework, which by inexactly solving the subproblems as well as employing non-positive curvature directions, are suitable for large scale and nonconvex settings. We obtain state-of-the-art convergence rates for various classes of non-convex problems and demonstrate competitive practical performance on a variety of problems.