An Optimal Algorithm for Strongly Convex Min-min Optimization

TL;DR

This paper introduces a new algorithm for strongly convex min-min optimization that reduces the computational complexity for gradient evaluations, especially when the condition number for one variable block is much larger than the other.

Contribution

The paper presents an algorithm that independently optimizes the gradient computation complexity for each variable block, outperforming existing methods in certain applications.

Findings

Reduces gradient computation complexity for x and y variables.

Outperforms existing methods when ppa_x ppa_y.

Effective in applications with asymmetric condition numbers.

Abstract

In this paper we study the smooth strongly convex minimization problem $\min_{x}\min_y f(x,y)$. The existing optimal first-order methods require $\mathcal{O}(\sqrt{\max\{\kappa_x,\kappa_y\}} \log 1/\epsilon)$ of computations of both $\nabla_x f(x,y)$ and $\nabla_y f(x,y)$, where $\kappa_x$ and $\kappa_y$ are condition numbers with respect to variable blocks $x$ and $y$. We propose a new algorithm that only requires $\mathcal{O}(\sqrt{\kappa_x} \log 1/\epsilon)$ of computations of $\nabla_x f(x,y)$ and $\mathcal{O}(\sqrt{\kappa_y} \log 1/\epsilon)$ computations of $\nabla_y f(x,y)$. In some applications $\kappa_x \gg \kappa_y$, and computation of $\nabla_y f(x,y)$ is significantly cheaper than computation of $\nabla_x f(x,y)$. In this case, our algorithm substantially outperforms the existing state-of-the-art methods.