Cubic regularized subspace Newton for non-convex optimization

TL;DR

This paper introduces a randomized coordinate second-order method called SSCN that uses cubic regularization in random subspaces to efficiently optimize high-dimensional non-convex functions, with proven convergence guarantees and practical speed-ups.

Contribution

It proposes a novel subspace cubic regularization method with theoretical convergence guarantees and adaptive sampling, improving efficiency in high-dimensional non-convex optimization.

Findings

Convergence guarantees for non-convex functions with subspace methods.

Complexity matches $ ilde{O}( ext{epsilon}^{-3/2})$ rate for increasing subspace size.

Experimental results show significant speed-ups over first-order methods.

Abstract

This paper addresses the optimization problem of minimizing non-convex continuous functions, which is relevant in the context of high-dimensional machine learning applications characterized by over-parametrization. We analyze a randomized coordinate second-order method named SSCN which can be interpreted as applying cubic regularization in random subspaces. This approach effectively reduces the computational complexity associated with utilizing second-order information, rendering it applicable in higher-dimensional scenarios. Theoretically, we establish convergence guarantees for non-convex functions, with interpolating rates for arbitrary subspace sizes and allowing inexact curvature estimation. When increasing subspace size, our complexity matches $\mathcal{O}(\epsilon^{-3/2})$ of the cubic regularization (CR) rate. Additionally, we propose an adaptive sampling scheme ensuring exact convergence rate of $\mathcal{O}(\epsilon^{-3/2}, \epsilon^{-3})$ to a second-order stationary point, even without sampling all coordinates. Experimental results demonstrate substantial speed-ups achieved by SSCN compared to conventional first-order methods.