Computing the Newton-step faster than Hessian accumulation

TL;DR

This paper introduces an algorithm that computes the Newton-step more efficiently by leveraging the function's computational graph, reducing complexity from cubic to a function of graph parameters, applicable to general optimization problems.

Contribution

It generalizes nonlinear optimal-control methods to broader optimization problems, achieving faster Newton-step computation based on graph decomposition.

Findings

Reduces Newton-step computation complexity from O(N^3) to O(mτ^3)

Applicable to dense Hessian cases with improved iteration complexity

Generalizes LQR-based methods to general optimization problems

Abstract

Computing the Newton-step of a generic function with $N$ decision variables takes $O(N^3)$ flops. In this paper, we show that given the computational graph of the function, this bound can be reduced to $O(m\tau^3)$, where $\tau, m$ are the width and size of a tree-decomposition of the graph. The proposed algorithm generalizes nonlinear optimal-control methods based on LQR to general optimization problems and provides non-trivial gains in iteration-complexity even in cases where the Hessian is dense.