Time-Varying Convex Optimization with $O(n)$ Computational Complexity

TL;DR

This paper introduces $O(n)$ algorithms for unconstrained time-varying convex optimization that efficiently track the minimizer by leveraging first-order derivatives, significantly reducing computational complexity from $O(n^3)$ to $O(n)$ and applicable to non-convex problems.

Contribution

The paper proposes novel $O(n)$ algorithms that utilize only first-order derivatives to solve time-varying convex optimization, reducing computational cost and broadening applicability.

Findings

Algorithms achieve $O(n)$ complexity per timestep.

Effective tracking of time-varying minimizers demonstrated.

Applicable to non-convex optimization problems.

Abstract

In this article, we consider the problem of unconstrained time-varying convex optimization, where the cost function changes with time. We provide an in-depth technical analysis of the problem and argue why freezing the cost at each time step and taking finite steps toward the minimizer is not the best tracking solution for this problem. We propose a set of algorithms that by taking into account the temporal variation of the cost aim to reduce the tracking error of the time-varying minimizer of the problem. The main contribution of our work is that our proposed algorithms only require the first-order derivatives of the cost function with respect to the decision variable. This approach significantly reduces computational cost compared to the existing algorithms, which use the inverse of the Hessian of the cost. Specifically, the proposed algorithms reduce the computational cost from $O(n^3)$ to $O(n)$ per timestep, where $n$ is the size of the decision variable. Avoiding the inverse of the Hessian also makes our algorithms applicable to non-convex optimization problems. We refer to these algorithms as $O(n)$-algorithms. These $O(n)$-algorithms are designed to solve the problem for different scenarios based on the available temporal information about the cost. We illustrate our results through various examples, including the solution of a model predictive control problem framed as a convex optimization problem with a streaming time-varying cost function.