Dual Space Preconditioning for Gradient Descent

TL;DR

This paper introduces a dual space preconditioning technique for gradient descent that guarantees convergence under generalized relative smoothness and strong convexity assumptions, improving conditioning for complex convex problems.

Contribution

It proposes a novel dual space preconditioning method that extends relative smoothness concepts, enabling improved convergence guarantees for gradient descent on challenging convex functions.

Findings

Guarantees convergence under dual relative smoothness.

Achieves linear convergence with a generalized condition number.

Demonstrates effectiveness on p-norm regression and exponential penalty minimization.

Abstract

The conditions of relative smoothness and relative strong convexity were recently introduced for the analysis of Bregman gradient methods for convex optimization. We introduce a generalized left-preconditioning method for gradient descent, and show that its convergence on an essentially smooth convex objective function can be guaranteed via an application of relative smoothness in the dual space. Our relative smoothness assumption is between the designed preconditioner and the convex conjugate of the objective, and it generalizes the typical Lipschitz gradient assumption. Under dual relative strong convexity, we obtain linear convergence with a generalized condition number that is invariant under horizontal translations, distinguishing it from Bregman gradient methods. Thus, in principle our method is capable of improving the conditioning of gradient descent on problems with non-Lipschitz gradient or non-strongly convex structure. We demonstrate our method on p-norm regression and exponential penalty function minimization.

Equations

$$\underset{x\in {\mathbb{R}}^d}{\min }f(x).$$

$$x_{i+1}=\underset{x\in \mathrm{dom}f}{\mathrm{argmin}}\left\{⟨\nabla f(x_i),x⟩+\frac{L}{2}{\Vert x-x_i\Vert }^2\right\},$$

$$\mu I⪯{\nabla }^{2}f(x)⪯LI\text{ for all }x\in \mathrm{int}(\mathrm{dom}f).$$

$$x_{i+1}=x_i-\frac{1}{L^{\ast }}\nabla k(\nabla f(x_i)).$$

$$x_{i+1}=\underset{x\in \mathrm{dom}f}{\mathrm{argmin}}\left\{⟨\nabla f(x_i),x⟩+L{D}_h(x,x_i)\right\}$$

$$\mu {\nabla }^{2}h(x)⪯{\nabla }^{2}f(x)⪯L{\nabla }^{2}h(x)\text{ for all }x\in \mathrm{int}(\mathrm{dom}f).$$

$${\mu }^{\ast }[{\nabla }^{2}k(\nabla f(x)){]}^{-1}⪯{\nabla }^{2}f(x)⪯L^{\ast }[{\nabla }^{2}k(\nabla f(x)){]}^{-1}\text{ for all }x\in \mathrm{int}(\mathrm{dom}f).$$

$$P^{-1}Ax=P^{-1}b\text{or}A{P}^{-1}y=b\text{ s.t. }x=P^{-1}y$$

$$\nabla k(\nabla f(x))=0\text{or}\nabla f(\nabla h^{\ast }(y))=0\text{ s.t. }x=\nabla h^{\ast }(y),$$

$$h^{\ast }(x^{\ast })=\sup \{⟨x,x^{\ast }⟩-h(x):x\in \mathrm{dom}h\}.$$

$$h(0)+\frac{\Vert x_i\Vert }{r}\left(h\left(\frac{r{x}_i}{\Vert x_i\Vert }\right)-h(0)\right)<h(0)+\left(h(x_i)-h(0)\right)$$

$$h(x_i)>h(0)+\frac{\Vert x_i\Vert }{r}\left(\underset{x\in \mathcal{S}}{\inf }h(x)-h(0)\right).$$

$$\nabla h^{\ast }(\nabla h(x))=x.$$

$${\nabla }^2{h}^{\ast }(\nabla h(x)){\nabla }^{2}h(x)={\nabla }^{2}h(x){\nabla }^2{h}^{\ast }(\nabla h(x))=I.$$

$$d_L(x)=Lg(x)-h(x)d_{\mu }(x)=h(x)-\mu g(x)$$

$$h^{\prime }(x;y)=\underset{ϵ\downarrow 0}{\lim }\frac{h(x+ϵy)-h(x)}{ϵ}.$$

$$D_h(x,y)=h(x)-h(y)-h^{\prime }(y;x-y).$$

$$r(t)\ge r(s)+r_{+}^{\prime }(s)(t-s).$$

$$h(x)-h(y)-\frac{h(x_t)-h(y)}{t}\le Lg(x)-Lg(y)-L\frac{g(x_t)-g(y)}{t}$$

$$D_{d_L}(x_s,x_t)=L{D}_g(x_s,x_t)-D_h(x_s,x_t)\ge 0.$$

$${\nabla }^{2}h(x)⪯L{\nabla }^{2}g(x)\forall x\in Q\setminus \{z\}.$$

$$\mu {\nabla }^{2}g(x)⪯{\nabla }^{2}h(x)\forall x\in Q\setminus \{z\}.$$

$$r^{\prime }(t)=r^{\prime }(a)+r^{\prime }(t)-r^{\prime }(a)=r^{\prime }(a)+\underset{s\to a}{\lim }{\int }_s^t⟨x-y,{\nabla }^2{d}_L(x_t)(x-y)⟩.$$

$$x_{i+1}=\arg \underset{x\in \mathrm{dom}f}{\min }\left\{⟨\nabla f(x_i)-L\nabla h(x_i),x⟩+Lh(x)\right\}.$$

$$D_k(\nabla f(y),\nabla f(x))\le L^{\ast }{D}_f(x,y)$$

$$D_{f^{\ast }}(\nabla f(y),\nabla f(x))\le D_f(x,y)$$

$$D_{f^{\ast }}(\nabla f(y),\nabla f(x))$$

$$\stackrel{(a)}{\le }{f}^{\ast }(\nabla f(y))-f^{\ast }(\nabla f(x))-⟨x,\nabla f(y)-\nabla f(x)⟩$$

$$\stackrel{(b)}{=}-f(y)+⟨\nabla f(y),y⟩+f(x)-⟨\nabla f(x),x⟩-⟨x,\nabla f(y)-\nabla f(x)⟩$$

$$=f(x)-f(y)+⟨\nabla f(y),y-x⟩=D_f(x,y)$$

$$D_k(\nabla f(y),\nabla f(x))\le L^{\ast }{D}_f(x,y).$$

$${\mu }^{\ast }{D}_f(x,y)\le D_k(\nabla f(y),\nabla f(x)).$$

Full text

1 Introduction

1.1 Setting and method

We study the minimization of a proper, closed, and essentially smooth convex function $f:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$,

[TABLE]

For unconstrained $f$, i.e., $\operatorname{dom}f=\{x\in\mathbb{R}^{d}:f(x)<\infty\}=\mathbb{R}^{d}$, essential smoothness is simply differentiability. For constrained $f$, essential smoothness is the assumption that $f$ is differentiable on $\operatorname{int}(\operatorname{dom}f)\neq\emptyset$ and that the norm of the gradient grows without bound, $\left\lVert\nabla\!\!\;f(x)\right\rVert\to\infty$, as $x$ approaches the boundary of the domain. Thus, a global minimizer $x_{\min}$ of $f$, if it exists, is in $\operatorname{int}(\operatorname{dom}f)$. The method that we introduce (Algorithm 1.1) is a non-linear generalization of linear left-preconditioning for gradient descent (see, e.g., [15, sect. 9.4]), and our analysis relies on recent generalizations of the typical Lipschitz gradient assumption [7]. For the sake of exposition, we will assume in the introduction that $f$ is twice continuously differentiable on $\operatorname{int}(\operatorname{dom}f)$, but this is not a requirement of our method.

In the analysis of first-order methods, it is standard to assume that the derivatives of $f$ at some order are globally bounded by constants. For example, consider the gradient descent method, whose iterates satisfy

[TABLE]

where $L>0$ and $x_{0}\in\operatorname{int}(\operatorname{dom}f)$. A classical analysis shows that the iterates of gradient descent converge linearly in $i$, i.e., $f(x_{i})-f(x_{\min})=\mathcal{O}(\lambda^{i})$ for $\lambda=1-\mu/L$, when $f$ is assumed to be $\mu>0$ strongly convex and $\nabla\!\!\;f$ is assumed to be $L$-Lipschitz continuous (typically called “smoothness”). Taken together for twice continuously differentiable $f$, these conditions are equivalent to the conditions that the eigenvalues of the Hessian matrix of second-order partial derivatives $\nabla^{2}\!\!\;f(x)$ are everywhere lower bounded by $\mu>0$ (strong convexity) and upper bounded by $L>0$ (smoothness),

[TABLE]

Analyses of first-order methods using only non-constant bounds on the derivatives of $f$ have recently been discovered [11, 7, 45, 42, 31]. In particular, [7] studied the following generalized gradient method that takes a designed essentially smooth, strictly convex reference function $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ with $\operatorname{int}(\operatorname{dom}f)\subseteq\operatorname{int}(\operatorname{dom}h)$. Given $x_{0}\in\operatorname{int}(\operatorname{dom}f)$, this method’s iterates satisfy

[TABLE]

where $L>0$, $\left\langle\cdot,\cdot\right\rangle$ is the Euclidean inner product, and $D_{h}(x,y)=h(x)-h(y)-\left\langle\nabla\!\!\;h(y),x-y\right\rangle$ for $x,y\in\operatorname{int}(\operatorname{dom}h)$. (3) is due to [34] and falls in a family of so-called Bregman gradient methods. A standard analysis of (3) (see, e.g., [8]) makes the “absolute” assumptions that $f$ is Lipschitz continuous and that $h$ is strongly convex. In contrast, consider the following “relative” conditions between $f$ and $h$, for $\mu\geq 0$ and $L>0$

[TABLE]

For twice continuously differentiable $f$, Bauschke et al. [7] first showed that (4) with $\mu=0$ is a sufficient assumption to guarantee the sublinear convergence of $f(x_{i})-f(x_{\min})$ in (3). Lu et al. [31] extended this analysis, and showed that (4) with $\mu>0$ is sufficient for the linear convergence of $f(x_{i})-f(x_{\min})$. Conditions (4) are “relative” in the sense that it is possible for (4) to hold for $f$ and $h$ that are both non-smooth or non-strongly convex. For example, [7] study a Poisson inverse objective whose derivatives of all orders are unbounded as $x\to 0$. They design an appropriate $h$, whose Hessian is also unbounded at [math], but which satisfies (4). Analyses of first-order methods using non-constant bounds on the derivatives of $f$ have been extended to non-convex $f$ [12, 23] continuous convex optimization [30], composite least-squares problems [24], symmetric non-negative matrix factorization [23], and the Sinkhorn algorithm [33]. Notably, relative smoothness conditions have also been used to justify fast implementations of third-order tensor methods [36].

The method that we introduce (Algorithm 1.1) exploits an application of these relative conditions in the dual space through an essentially smooth, strictly convex dual reference function $k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ with $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$ and $0=\arg\min_{x^{*}}k(x^{*})$. The method is a generalization of left-preconditioned gradient descent, which we discuss in more detail in section 1.2. In section 3 we consider the conditions under which we can provide convergence rates for our method. For twice continuously differentiable $f$ sufficient conditions that we study are the existence of $\mu^{*}\geq 0$, $L^{*}>0$ such that

[TABLE]

When $\mu^{*}=0$, we show that $k(\nabla\!\!\;f(x_{i}))-k(0)$ converges sub-linearly with rate $\mathcal{O}(i^{-1})$ (and thus $x_{i}\to x_{\min}$) along the iterates of Algorithm 1.1. When $f$ is strictly convex and $\mu^{*}>0$, we show that $f(x_{i})-f(x_{\min})$ converges linearly with rate $\lambda^{*}=1-\mu^{*}/L^{*}$. As we show in section 3, assumptions (5) are relative smoothness and strong convexity assumptions in the dual space, and they are distinct from (4). In section 4, we design dual reference functions for $p$-norm regression (see [17, 2] and references therein) and exponential penalty functions (see, e.g., [21, 20]).

1.2 Preconditioning

In this paper, we introduce a generalization of linear left-preconditioning, which is a fundamental technique used in algorithms for solving linear systems. In this subsection, we review linear preconditioning, following closely Wathen’s short introduction [46], and give an interpretation of our method and Bregman gradient methods as left- and right-preconditioning, respectively.

Consider the problem of minimizing a positive-definite quadratic, which is equivalent to finding the solution $x$ of a linear system of $d$ equations with $d$ unknowns: $Ax=b$ where $b\in\mathbb{R}^{d}$ and $A\in\mathbb{R}^{d\times d}$ is symmetric and positive-definite. “Preconditioning” refers to the idea of modifying this system in a way that preserves the solution, but improves the convergence of iterative methods. For example, given a positive-definite $P\in\mathbb{R}^{d\times d}$, we may consider the following systems (known as left- or right-preconditioning, respectively):

[TABLE]

These have the same solution as the original, and if $P^{-1}A$ or $AP^{-1}$ approximates the identity, then iterative methods will converge faster. Indeed, for iterates of the conjugate gradients method (CG) [26], $\left\langle x-x_{i},A(x-x_{i})\right\rangle$ converges linearly with a rate that varies monotonically with the condition number $\kappa^{A}=\lambda_{\max}^{A}/\lambda_{\min}^{A}$, i.e., the ratio of the largest to the small eigenvalue of $A$ [25, Chap. 3.1]. Smaller condition number is better, so if $\kappa^{A}\gg\kappa^{P^{-1}A}$, then left-preconditioned CG will converge faster. Preconditioned methods typical solve a system with $P$ at every iteration. Thus, $P$ should satisfy two criteria: $\kappa^{P^{-1}A}$ should be small and $Px=b$ should be easy to solve. It may seem difficult to strike this balance, but it is possible in many cases. Wathen [46] gives an example due to Strang for Toeplitz matrices that reduces the complexity of linear solves from $\mathcal{O}(d^{2})$ to $\mathcal{O}(d\log d)$. More generally, preconditioners are considered essential in solvers for very large, sparse, linear systems [10, 39].

Consider now the more general problem (P) for an unconstrained $f$. One can show that the following are stationary conditions of Algorithm 1.1 or (3), respectively:

[TABLE]

where $\nabla\!\!\;h^{*}(y)=\arg\max_{x\in\mathbb{R}^{d}}\left\langle x,y\right\rangle-h(x)$. Clearly, (6) specializes (7) for appropriately chosen quadratic $f,k,h$. Thus, our method and the Bregman gradient method (3) may be seen as a generalization of left- and right-preconditioning for gradient descent, respectively. Moreover, for symmetric, positive-definite $A,P\in\mathbb{R}^{d\times d}$, the existence of $L,\mu>0$ such that $\mu P\preceq A\preceq LP$ guarantees $\kappa^{P^{-1}A}\leq L/\mu$ and an error bound on preconditioned CG. This is generalized by the primal (4) and dual (5) relative conditions. However, in contrast to the linear case, the choice of left (dual) vs. right (primal) in the non-linear case is much more consequential and the two methods are not equivalent in general (left- and right-preconditioning for CG are equivalent [39, Chap. 9.1]). The class of $f$ satisfying the dual conditions (5) for a fixed $k$ is closed under horizontal translations. This is not true in general for $f$ satisfying the primal conditions (4) for a fixed $h$. Thus, in general, $\mu\neq\mu^{*}$, $L\neq L^{*}$, and the global information encoded in the dual reference function $k$ is distinct from the information encoded in the reference function $h$.

Non-linear preconditioning is far less studied, but has been considered in a number of works. Non-linear preconditioning methods have recently been shown to stabilize Euler discretization schemes of stochastic differential equations [27, 40]. In fact, the non-linear preconditioning of [27] is the same as the one we consider for exponential penalty functions. Finally, recent work [18, 22] developed non-linear preconditioning schemes for Newton’s method applied to problems arising from the discretization of partial differential equations.

2 Convex analysis background

2.1 Essential smoothness and convex conjugates

In this section we review some basic facts of convex analysis that will be used throughout. Let $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be a proper closed convex function with domain $\operatorname{dom}h=\{x:\mathbb{R}^{d}:h(x)<\infty\}$. To indicate $\operatorname{dom}h=\mathbb{R}^{d}$, we simply define $h:\mathbb{R}^{d}\to\mathbb{R}$ as ranging only over the reals. $\partial h(x)$ denotes the subdifferential of $h$ at $x\in\mathbb{R}^{d}$. For a proper convex functions, being closed is equivalent to being lower semi-continuous (lsc). Let $\left\lVert\cdot\right\rVert$ and $\left\langle\cdot,\cdot\right\rangle$ indicate the Euclidean norm and inner product, respectively, unless otherwise specified. The convex conjugate $h^{*}:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ of a proper closed convex function $h$ is given by

[TABLE]

$h^{*}$ is also a proper closed convex function, and $(h^{*})^{*}=h$ [38, Cor. 12.2.1]. For more on $h^{*}$, we refer readers to [38, 15, 13].

In this work, we study the minimization of an essentially smooth convex function $f$ [38, Chap. 25], which can be thought of as an assumption of differentiability. For constrained $f$, essential smoothness comes with additional structure that prevents $f$ from having sharp edges at the boundary of its domain. In some cases, we will consider the additional assumption that $f$ is strictly convex on the interior of its domain.

Definition 2.1** (Essential smoothness and Legendre convexity).**

Let $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be a proper closed convex function. $h$ is essentially smooth if,

$\operatorname{int}(\operatorname{dom}h)$* is not empty.* 2. 2.

$h$* is differentiable on $\operatorname{int}(\operatorname{dom}h)$, with $\lim_{i\to\infty}\left\lVert\nabla\!\!\;h(x_{i})\right\rVert=\infty$ whenever $x_{i}\in\operatorname{int}(\operatorname{dom}h)$ is a sequence converging to the boundary of $\operatorname{int}(\operatorname{dom}h)$.*

$h$* is Legendre convex, if additionally*

$h$* is strictly convex on $\operatorname{int}(\operatorname{dom}h)$*

In this work, the assumption that $h$ is essentially smooth carries with it the implied assumption that $h$ is proper and closed.

Essentially smooth convex functions can only be minimized in their interior.

Lemma 2.2**.**

If $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ is an essentially smooth convex function that is minimized at $x_{\min}\in\operatorname{dom}h$, then $x_{\min}\in\operatorname{int}(\operatorname{dom}h)$.

Proof.

Suppose that $x_{\min}$ is a boundary point. Since $\operatorname{int}(\operatorname{dom}h)\neq\emptyset$, by convexity there exists a line segment connecting the boundary point $x_{\min}$ and any other interior point $a$. However, by [38, Lem. 26.2], we know that the directional derivative converges to $-\infty$ as we tend towards the boundary point on this line segment, hence $x_{\min}$ could not be a minimum of $h$. ∎

Legendre convex functions (essentially smooth, strictly convex functions) have even more convenient structure. One consequence of Legendre structure, which will be used in our analysis to show that $k$ is radially unbounded, is that achieving a minimum is sufficient to imply that a Legendre convex function grows without bound.

Lemma 2.3**.**

Let $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be a Legendre convex function that is minimized at $0\in\operatorname{dom}h$. Then $h$ is radially unbounded, i.e., if $x_{i}\in\mathbb{R}^{d}$ is a sequence such that $\left\lVert x_{i}\right\rVert\to\infty$, then $h(x_{i})\to\infty$.

Proof.

First, by Lemma 2.2 it follows that $0\in\operatorname{int}(\operatorname{dom}h)$. Because $h$ is strictly convex, [math] is the unique minimum of $h$. Thus, we can define the sphere $\mathcal{S}=\{x\in\mathbb{R}^{d}:\left\lVert x\right\rVert=r\}$ for some $r>0$ such that $\mathcal{S}\in\operatorname{int}(\operatorname{dom}h)$. By continuity of $h$ in the interior of its domain, and the uniqueness of the minimum at zero, we have $\inf_{x\in\mathcal{S}}h(x)>h(0)$. Now, assume without loss of generality that $\left\lVert x_{i}\right\rVert>r$. By strict convexity of Legendre functions, property 3, we have

[TABLE]

and thus

[TABLE]

Our result follows by taking $i\to\infty$. ∎

A second key consequence of Legendre structure is that the gradient map $\nabla\!\!\;h$ is invertible and given by $(\nabla\!\!\;h)^{-1}=\nabla\!\!\;h^{*}$, which also gives a characterization of the inverse of $\nabla^{2}\!\!\;h(x)$. We summarize both of these properties in Lemma 2.4.

Lemma 2.4**.**

Let $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be Legendre convex. Then, $h^{*}$ is Legendre convex, the map $\nabla\!\!\;h$ is one-to-one and onto from the open set $\operatorname{int}(\operatorname{dom}h)$ onto the open set $\operatorname{int}(\operatorname{dom}h^{*})$, continuous in both directions, and for all $x\in\operatorname{int}(\operatorname{dom}h)$

[TABLE]

If $h$ is $C^{2}$ on an open set containing $x$ and $\det\nabla^{2}\!\!\;h(x)\neq 0$, then

[TABLE]

Proof.

For the first part see Rockafellar [38, Thm. 26.5]. For (12), note that, by the inverse function theorem, $\nabla\!\!\;h^{*}$ is continuously differentiable at $\nabla\!\!\;h(x)$ under the assumption that $\nabla\!\!\;h$ is continuously differentiable on an open set containing $x$. The remainder follows by the chain rule applied to (11). ∎

2.2 Relative smoothness and relative strong convexity

Analyses of first-order methods for differentiable optimization typically require that $\nabla\!\!\;f$ is Lipschitz continuous (smooth). Recent generalizations of smoothness (and strong convexity) [7, 31] can be used to guarantee convergence of first-order methods beyond the Lipschitz $\nabla\!\!\;f$ case. We will use these in our analysis of dual preconditioning. Following [7], we define these relative conditions in terms of zeroth-order properties.

Definition 2.5** (Relative smoothness and strong convexity).**

Let $h,g:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be proper closed convex functions, $Q\subseteq\operatorname{dom}h\cap\operatorname{dom}g$ be a convex set, and $L,\mu\geq 0$. Define $d_{L},d_{\mu}:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ for $x\in Q$ by

[TABLE]

and for $x\notin Q$ by $d_{L}(x)=d_{\mu}(x)=\infty$. $h$ is $L$-smooth relative to $g$ on $Q$, if $d_{L}$ convex. $h$ is $\mu$-strongly convex relative to $g$ on $Q$, if $d_{\mu}$ is convex.

The special cases with $g(x)=\left\lVert x\right\rVert_{2}^{2}/2$ are exactly the classical conditions of strong convexity and smoothness. We now provide first- and second-order characterizations.

2.3 First-order characterizations for relative conditions

The first-order characterizations of relative smoothness and strong convexity are given in terms of the Bregman divergence [16, 6], which for essentially smooth convex $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ and $x\in\operatorname{dom}h,y\in\operatorname{int}(\operatorname{dom}h)$ is given by $h(x)-h(y)-\left\langle\nabla h(y),x-y\right\rangle$. Unfortunately, in our analysis, we will require smoothness relative to $f^{*}$, which can fail to be differentiable when $f$ is essentially smooth. Thus, we will make use of a generalization of the Bregman divergence, which we define via the one-sided directional derivative of $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{-\infty,\infty\}$ with respect to $y\in\mathbb{R}^{d}$ at a point $x$ where $h$ is finite:

[TABLE]

This may take values in $\{\infty,-\infty\}$. The advantage of one-sided direction derivatives is that they always exist at $x\in\operatorname{dom}h$ for proper convex $h$ [38, Thm. 23.1]. We are now prepared to define a novel generalization of the Bregman divergence.

Definition 2.6** (Generalized Bregman divergences).**

Let $h:\mathbb{R}^{d}\to\mathbb{R}\cup\{-\infty,\infty\}$ be a function. Let $x,y\in\mathbb{R}^{d}$ be points at which $h$ is finite and $h^{\prime}(y;x-y)$ exists. Define the generalized Bregman divergence,

[TABLE]

If $h$ is proper, closed, and convex, then $D_{h}(x,y)$ is defined for all $x,y\in\operatorname{dom}h$ [38, Thm. 23.1] and is finite for $y\in\operatorname{dom}\partial h=\{x\in\mathbb{R}^{d}:\partial h(x)\neq\emptyset\}$ [38, Thm. 23.2].

Clearly, $D_{h}(x,y)$ coincides with the standard Bregman divergence if $h$ is differentiable at $y$. The advantage of the generalization is that it allows us to define the relative conditions in terms of first-order properties without the assumption of differentiability.

Proposition 2.7** (First-order characterizations of relative conditions).**

Let $h,g:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be proper closed convex functions, $Q\subseteq\operatorname{dom}h\cap\operatorname{dom}g$ be a convex set, and $L,\mu\geq 0$. The following are equivalent

$h$* is $L$-smooth relative to $g$ on $Q$.* 2. 2.

For all $x,y\in Q$, $D_{h}(x,y)\leq LD_{g}(x,y)$.

The following are equivalent

$h$* is $\mu$-strongly convex relative to $g$ on $Q$.* 2. 4.

For all $x,y\in Q$, $\mu D_{g}(x,y)\leq D_{h}(x,y)$.

To prove these equivalences, we will use two lemmas, which extend the first-order characterization of one-dimensional convexity to the non-differentiable case.

Lemma 2.8** (A variant of the mean value theorem).**

Let $r\colon[0,1]\to\mathbb{R}$ be a continuous function. Define $r^{\prime}_{+}(z)=\lim_{\epsilon\downarrow 0}(r(z+\epsilon)-r(z))/\epsilon$. Assuming that $r^{\prime}_{+}(z)$ exists for $z\in[0,1)$, if $s,t\in[0,1]$ and $s<t$, then there is exists $z\in[s,t)$ such that $r^{\prime}_{+}(z)\geq(r(t)-r(s))/(t-s)$.

Proof.

We can add a linear function to $r$ without changing the difference between the two sides on the inequality, so without loss of generality we may assume that $r(s)=r(t)=0$. Then, we want to prove that $r^{\prime}_{+}(z)\geq 0$ for some $z\in[s,t]$. Since $r$ is continuous, it has a minimum in $[s,t]$, so there is a $z\in[s,t]$ such that $r(u)\geq r(z)$ for every $u\in[s,t]$. If $z=t$, then $r(z)=r(s)$, so we could instead take $z=s$. Thus we may assume that $z\in[s,t)$. Then $r^{\prime}_{+}(z)=\lim_{u\downarrow z,\,u\in(z,t)}\frac{r(u)-r(z)}{u-z}\geq 0$, because $r(u)\geq r(z)$, proving the claim. ∎

Lemma 2.9** (Characterization of one-dimensional convexity).**

Let $r\colon[0,1]\to\mathbb{R}$ be a continuous function. $r$ is convex on $[0,1]$ if and only if $r^{\prime}_{+}(z)$ (defined in Lemma 2.8) exists for all $z\in(0,1)$, and for all $s,t\in(0,1)$ such that $s<t$,

[TABLE]

Proof.

Suppose that $r$ is not convex on $[0,1]$. Then by continuity, it is also not convex on $(0,1)$. After adding a linear function, we can arrange that $0=r(s)=r(t)<r(z)$ for some $0<s<z<t<1$. Since $r$ is continuous, $r$ achieves its maximum restricted to the interval $[s,t]$, so we could choose $z\in(s,t)$ such that $r(z)=\max_{u\in[s,t]}r(u)>0$. Since $r(z)>0$ and $r$ is continuous, there is a $u\in(s,z)$ such that $r(v)>0$ for every $v\in[u,z]$. By Lemma 2.8, there is a $v\in[u,z)$ such that $r_{+}^{\prime}(v)\geq\frac{r(z)-r(u)}{z-u}\geq 0$, and so $0=r(t)\geq r(v)+r^{\prime}_{+}(v)(t-v)\geq r(v)>0$. This contradiction proves that $r$ is indeed convex.

Now suppose that $r$ is convex and continuous on $[0,1]$. By [38, Thm. 23.1] the difference quotient $\frac{r(z+\epsilon)-r(z)}{\epsilon}$ is a non-decreasing function of $\epsilon$ for $\epsilon>0$ and $z\in[0,1)$, and limit $r^{\prime}_{+}(z)$ exists. Using the fact that the difference quotient is non-decreasing in $\epsilon$ it follows that $r(t)\geq r(s)+r^{\prime}_{+}(s)(t-s)$ for every $s,t\in[0,1]$, $s<t$. ∎

We are now prepared to provide the proof of equivalence between the zeroth- and first-order definitions of the relative conditions.

Proof of Proposition 2.7.

We only show the equivalence of 1. and 2., the proof of the equivalence of 3. and 4. is similar. First, suppose that 1. holds, i.e., $d_{L}$ is convex. Let $x,y\in Q$. Then for $x_{t}=y+t(x-y)$, we have (after dividing by $t$ and rearranging)

[TABLE]

Taking the limit $t\downarrow 0$ gives us that $D_{h}(x,y)\leq LD_{g}(x,y)$, with the existence of the limits following from [38, Thm. 23.1].

For the other direction, suppose that $D_{h}(x,y)\leq LD_{g}(x,y)$ for every $x,y\in Q$. Let $x,y\in Q$, and $x_{t}=y+t(x-y)$. Then for any $0<s<t<1$, it is easy to check that both $h^{\prime}(x_{t};x_{s}-x_{t})$ and $g^{\prime}(x_{t};x_{s}-x_{t})$ are finite (if one of the directional derivatives is non-finite, then this would contradict the convexity or finiteness of these functions over $Q$). Thus $D_{h}(x_{s},x_{t})$ and $D_{g}(x_{s},x_{t})$ are finite and satisfy $D_{h}(x_{s},x_{t})\leq LD_{g}(x_{s},x_{t})$ . Thus, it follows that for any $0<s<t<1$,

[TABLE]

Let $r(t)=d_{L}(x_{t})$ for $t\in[0,1]$ be the restriction of $d_{L}$ on the line segment between $x$ and $y$, then (18) implies that the condition (16) holds. $r$ is a continuous function by [38, Thm. 10.2]. Thus, by Lemma 2.9, $r$ is a convex function on $[0,1]$. This holds for all $x,y\in Q$, thus $d_{L}$ is convex.

∎

2.4 Second-order characterizations of relative conditions

Verifying relative smoothness or strong convexity is typically done via second-order conditions. Just as Lipschitz continuity of $\nabla\!\!\;h$ can be characterized by a bound on $\nabla^{2}\!\!\;h$, the relative conditions can be characterized by the second derivatives of $h$ and $g$ [7, 31]. Proposition 2.10 allows $\nabla^{2}\!\!\;h,\nabla^{2}\!\!\;g$ to be undefined at a point, a slight generalization of the standard result that is useful in our analysis when $\nabla^{2}\!\!\;f$ is undefined at $x_{\min}$.

Proposition 2.10** (Second-order characterizations of relative conditions).**

Let $h,g:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be proper closed convex functions that are differentiable on the interior of their domains. Let $Q\subseteq\operatorname{int}(\operatorname{dom}g)\cap\operatorname{int}(\operatorname{dom}h)$ be an open convex set, $z\in Q$, and $L,\mu\geq 0$. If $h,g$ are $C^{2}$ on $Q\setminus\{z\}$, then

$h$* is $L$-smooth relative to $g$ on $Q$ if and only if,*

[TABLE] 2. 2.

$h$* is $\mu$-strongly convex relative to $g$ on $Q$ if and only if,*

[TABLE]

Proof.

Again, we prove the relative smoothness equivalence, and relative strong convexity follows similarly. For relative smoothness, $(\Rightarrow)$ follows from part one of [35, Thm. 2.1.4] applied to $d_{L}$ at $x\in Q$. For $(\Leftarrow)$, it is sufficient to prove convexity of the restriction of $d_{L}$ to an open line segment with endpoints $x,y\in Q$. Let $x_{t}=y+t(x-y)$ and $r(t)=d_{L}(x_{t})$ for $t\in(0,1)$. Let $a\in(0,1)$ be such that $x_{a}=z$, if it exists, or some arbitrary $a\in(0,1)$, otherwise. $d_{L}$ is continuously differentiable at all $x\in Q$ by [38, Thm 25.5]. Thus $r^{\prime}(t)=\left\langle\nabla\!\!\;d_{L}(x_{t}),x-y\right\rangle$ is a continuous and finite function of $t\in(0,1)$. If $r^{\prime}$ is non-decreasing, then the argument of [38, Thm. 4.4] gives us our result. Thus, with a slight abuse of notation,

[TABLE]

The limit is actually a one-sided limit, depending on $t\leq a$ or $t>a$. Either way, $\nabla^{2}\!\!\;d_{L}(x_{t})=L\nabla^{2}\!\!\;g(x_{t})-\nabla^{2}\!\!\;h(x_{t})$ is positive semi-definite, so $r^{\prime}$ is non-decreasing. ∎

3 Analysis of the dual preconditioned scheme

3.1 Motivation and assumptions

Relative smoothness of $f$ with respect to a reference function $h$ is the key assumption under which [7, 42, 31] analyzed the convergence of Bregman gradient methods. We now build towards an analysis of dual space preconditioned gradient descent method (Algorithm (1.1)) using the assumption that $k$ is smooth relative to $f^{*}$. As shorthand to distinguish these two assumptions, we use the terms primal relative smoothness to refer to the condition that $f$ is $L$-smooth relative to $h$ and dual relative smoothness to refer to the condition that $k$ is $L^{*}$-smooth relative to $f^{*}$. To motivate our assumption, consider the following idealizations.

Consider the Bregman gradient method update (3), which can be rewritten as,

[TABLE]

In this form, it is clear that, if $h=f$ and $L=1$, then the iteration would converge in a single step to the minimizer of $h=f$. This is an idealization, because a single iteration would be as expensive to compute as the original problem. The spirit behind primal relative smoothness is that the condition $h=f$ can be relaxed to admit $h$ for which the update (19) is efficiently solvable and the iterates still converge.

Now, consider the case that $f$ is Legendre convex with a minimum at $x_{\min}$, and let $f_{c}^{*}(x^{*})=f^{*}(x^{*})-\left\langle x^{*},x_{\min}\right\rangle$ for $x^{*}\in\mathbb{R}^{d}$. Notice that $\nabla\!\!\;f_{c}^{*}(\nabla\!\!\;f(x))=x-x_{\min}$ by Lemma 2.4 and that Algorithm 1.1 with $k=f_{c}^{*}$ and $L_{i}=1$ would converge in a single step to the minimizer $x_{\min}$ of $f$. Thus, in analogy to the relative smoothness analysis of [7] in the primal space, the spirit behind our analysis under dual relative smoothness is that the requirement $k=f_{c}^{*}$ can be relaxed while maintaining the convergence of Algorithm 1.1. In particular, sufficient assumptions on $k$ are that it is minimized at [math] and smooth relative to $f^{*}$.

More precisely, our analysis of Algorithm 1.1 uses following assumptions.

Assumption 3.1**.**

$f:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$* is convex and essentially smooth.* 2. 2.

$k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$* is Legendre convex and uniquely minimized at [math].* 3. 3.

$\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$* and for all $x,y\in\operatorname{int}(\operatorname{dom}f)$,*

[TABLE]

As we show in the following sections, Assumption 3.1.3 is a necessary condition of the relative smoothness of $k$ with respect to $f^{*}$. Assumption 3.1.3 is the assumption that requires the most effort to verify, since the convexity of $L^{*}f^{*}-k$ will typically be difficult to check. For this reason, we also provide second-order sufficient conditions expressed (mostly) in terms of conditions on $f$ and $k$.

3.2 Dual relative conditions for Legendre convex objectives

When $f$ is essentially smooth and strictly convex (Legendre), we are able to provide clean characterizations of the dual relative conditions. In particular, Assumption 3.1.3 is necessary and sufficient for the smoothness of $k$ relative to $f^{*}$ on $\operatorname{int}(\operatorname{dom}f^{*})$. We begin by linking $D_{f}$ and $D_{f^{*}}$ in what is a well-known identity for Legendre convex $f$.

Lemma 3.2**.**

If $f:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ is an essentially smooth convex function, then

[TABLE]

for all $x,y\in\operatorname{int}{(\operatorname{dom}f)}$. If $f$ is Legendre convex, then this is an equality.

Proof.

Note, by [38, Cor. 26.4.1], we have $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))=\operatorname{dom}\partial f^{*}$. Thus, $D_{f^{*}}(\nabla\!\!\;f(y),\nabla\!\!\;f(x))$ is finite for any $x,y\in\operatorname{int}(\operatorname{dom}f)$. Note $x\in\partial f^{*}(\nabla\!\!\;f(x))$. Now,

[TABLE]

where $(a)$ follows from [38, Thm. 23.2], $(b)$ follows from [38, Thm. 26.4]. If $f$ is Legendre convex, then by Lemma 2.4, $f^{*}$ is Legendre, $f^{*}$ is differentiable on $\operatorname{int}(\operatorname{dom}f^{*})=\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))$, and $(a)$ is an equality [38, Thm. 23.4] . ∎

We can now provide first-order characterizations of the dual relative conditions.

Proposition 3.3** (First-order characterization of dual relative conditions, Legendre convex case).**

Let $f,k:\mathbb{R}^{d}\to\{\mathbb{R},\infty\}$ be Legendre convex functions. The following are equivalent.

$k$* is $L^{*}$-smooth relative to $f^{*}$ on $\operatorname{int}(\operatorname{dom}f^{*})$.* 2. 2.

$\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$, and for all $x,y\in\operatorname{int}(\operatorname{dom}f)$,

[TABLE]

The following are equivalent.

$k$* is $\mu^{*}$-strongly convex relative to $f^{*}$ on $\operatorname{int}(\operatorname{dom}f^{*})$.* 2. 4.

$\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$, and for all $x,y\in\operatorname{int}(\operatorname{dom}f)$,

[TABLE]

Proof.

We prove the relative smoothness results, and the relative strong convexity ones follow similarly. First, notice that that $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))=\operatorname{int}(\operatorname{dom}f^{*})$. by Lemma 2.4. For (1 $\Rightarrow$ 2), by the definition of relative smoothness, we have $\operatorname{int}(\operatorname{dom}f^{*})\subseteq\operatorname{dom}k$, and since this is an open set, we necessarily have $\operatorname{int}(\operatorname{dom}f^{*})\in\operatorname{int}(\operatorname{dom}k)$. By Proposition 2.7 we have that for all $x^{*},y^{*}\in\operatorname{int}(\operatorname{dom}f^{*})$,

[TABLE]

By Lemmas 2.4 and 3.2, this implies

[TABLE]

for all $x,y\in\operatorname{int}(\operatorname{dom}f)$. For (2 $\Rightarrow$ 1), by Lemma 3.2, we have for all $x,y\in\operatorname{int}(\operatorname{dom}f)$,

[TABLE]

Using this, Proposition 2.7 implies that $k$ is $L^{*}$-smooth relative to $f^{*}$ on $\operatorname{int}(\operatorname{dom}f^{*})$. ∎

If $f$ is Legendre convex, then the dual relative conditions have a natural second-order characterization, which reveals the structure of the difference between them and primal relative conditions. Again, typically it is easiest to prove dual relative smoothness (or strong convexity) via these second-order conditions.

Proposition 3.4** (Second-order characterizations of dual relative conditions, Legendre convex case).**

Let $f:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be Legendre convex, minimized at $x_{\min}$, and $C^{2}$ on $\operatorname{int}(\operatorname{dom}f)\setminus\{x_{\min}\}$ such that $\det\nabla^{2}\!\!\;f(x)\neq 0$ at $x\in\operatorname{int}(\operatorname{dom}f)\setminus\{x_{\min}\}$. Let $k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be Legendre convex, $C^{2}$ on $\operatorname{int}{(\operatorname{dom}f^{*})}\setminus\{0\}$ such that $\det\nabla^{2}\!\!\;k(x^{*})\neq 0$ at $x^{*}\in\operatorname{int}(\operatorname{dom}f^{*})\setminus\{0\}$. Let $L,\mu\geq 0$.

$k$* is $L^{*}$-smooth relative to $f^{*}$ on $\operatorname{int}(\operatorname{dom}f^{*})$ if and only if,*

[TABLE] 2. 2.

$k$* is $\mu^{*}$-strongly convex relative to $f^{*}$ on $\operatorname{int}(\operatorname{dom}f^{*})$ if and only if,*

[TABLE]

Remark 3.5*.*

It is well-known that the primal and dual relative conditions are equivalent in the case of $\nabla^{2}\!\!\;h(x)=I=\nabla^{2}\!\!\;k(x^{*})$ (see, e.g., [48, 41, 28]). In particular, if $f$ is $\mu$-strongly convex and $L$-smooth on $\operatorname{int}(\operatorname{dom}f)$, then its convex conjugate $f^{*}$ is $(1/L)$-strongly convex and $(1/\mu)$-smooth on $\operatorname{int}(\operatorname{dom}f^{*})$. In fact, for twice continuously differentiable $f$, the equivalence is a simple consequence of Propositions 2.10 and 3.4. However, this equivalence is not true in general.

Given a Legendre convex $g:\mathbb{R}\to\mathbb{R}\cup\{\infty\}$ define the following sets of functions

[TABLE]

Let $k(x^{*})=|x^{*}|^{q}/q$ for $x^{*}\in\mathbb{R}$ and $1<q<2$. A simple argument by contradiction shows that $\mathscr{F}_{k}^{*}\nsubseteq\mathscr{F}_{h}$ for all twice continuously differentiable $h:\mathbb{R}\to\mathbb{R}$, implying that the primal and dual relative conditions are not equivalent in general. Consider

[TABLE]

for $p=\frac{q}{q-1}$ and $x\in\mathbb{R}$. First $f_{b}\in\mathcal{F}_{k}^{*}$ for all $b$, which follows from $[k^{\prime\prime}(f^{\prime}_{b}(x))]^{-1}=(p-1)|x-b|^{p-2}=f_{b}^{\prime\prime}(x)$ and Proposition 3.4. On the other hand, suppose there is some twice continuously differentiable $h:\mathbb{R}\to\mathbb{R}$ such that $f_{b}\in\mathscr{F}_{h}$ for all $b$. Then there exists $\mu>0$ such that $\mu h^{\prime\prime}(b)\leq f_{b}^{\prime\prime}(b)=0$ for all $b$. This implies that $h^{\prime\prime}(x)\equiv 0$ and thus $h(x)\equiv 0$. However, this leads to a contradiction, because smoothness is violated: $f_{b}^{\prime\prime}(b+\epsilon)>0=Lh^{\prime\prime}(x)$ for any $L,\epsilon>0$.

Proof of Proposition 3.4.

We prove the relative smoothness result, and the relative strong convexity one follows similarly. By Lemma 2.4, if $\nabla\!\!\;f$ is continuously differentiable for $x\in\operatorname{int}(\operatorname{dom}f)\setminus\{x_{\min}\}$, then $\nabla\!\!\;f^{*}$ is continuously differentiable for $x^{*}\in\operatorname{int}(\operatorname{dom}f^{*})\setminus\{0\}$ by the inverse function theorem. Thus, by Proposition 2.10 dual relative smoothness is equivalent to: for all $x^{*}\in\operatorname{int}(\operatorname{dom}f^{*})\setminus\{0\}$,

[TABLE]

By Lemma 2.4, (27) is equivalent to for all $x\in\operatorname{int}(\operatorname{dom}f)\setminus\{x_{\min}\}$,

[TABLE]

Since $A^{-1}\preceq B^{-1}$ is equivalent to $B\preceq A$ for positive definite matrices, we are done. ∎

A major difference between the primal and dual relative conditions is the fact that dual relative conditions are invariant under horizontal translations of $f$. To see why, let $k$ be $L^{*}$-smooth relative to $f^{*}$ on a convex set $Q$. Define $g(x)=f(x-z)$ for $z\in\mathbb{R}^{d}$. Then, by [38, Thm. 12.3], $g^{*}(x^{*})=f^{*}(x^{*})+\left\langle z,x^{*}\right\rangle$. Bregman divergences of functions that differ only in affine terms are identical (see [6] for the differentiable case), so we have for all $x^{*},y^{*}\in Q$, $D_{k}(x^{*},y^{*})\leq L^{*}D_{f^{*}}(x^{*},y^{*})=L^{*}D_{g^{*}}(x^{*},y^{*})$. Thus $k$ is $L^{*}$-smooth relative to $g^{*}$ on $Q$. Invariance under horizontal translation is clearly easy to violate in the case of primal relative smoothness.

Even if $h$ is allowed to translate with $f$, the primal and dual relative conditions can lead to distinct conditioning. Given a positive definite $A\succ 0$, let

[TABLE]

for $1/p+1/q=1$ and $p>2$. It can be shown that $f$ satisfies both the dual (with respect to $k$) and primal (with respect to $h$) relative conditions. Nonetheless, the condition numbers are distinct. A simple calculation reveals that in this case

[TABLE]

where $\sigma_{\min}$ and $\sigma_{\max}$ are the smallest and largest singular values of $A$, respectively. Thus, the primal condition number is larger than the dual number (since $4-q=3-(p-1)^{-1}<p$ when $p>2$). Similarly, the example $f(x)=\|Ax-b\|_{4}^{4}/4+\|Cx-d\|_{2}^{2}/2$ of [31, p. 339] can be shown to have better conditioning under the dual preconditioned method than under the Bregman gradient method.

3.3 Dual relative conditions for essentially smooth objectives

We now show that the smoothness of $k$ relative to $f^{*}$ on $\operatorname{dom}f^{*}$ is a sufficient condition for Assumption 3.1.3. We also provide a sufficient, second-order condition.

Proposition 3.6**.**

Let $f,k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be essentially smooth convex functions. If $k$ is $L^{*}$-smooth relative to $f^{*}$ on $\operatorname{dom}f^{*}$, then $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$ and for all $x,y\in\operatorname{int}(\operatorname{dom}f)$,

[TABLE]

Proof.

Note that by [38, Cor. 26.4.1], $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))=\operatorname{dom}\partial f^{*}\subseteq\operatorname{dom}f^{*}$. We have $\operatorname{dom}f^{*}\subseteq\operatorname{dom}k$ by the definition of relative smoothness. By Proposition 2.7 and Lemma 3.2, for every $x,y\in\operatorname{int}(\operatorname{dom}f)$ we have

[TABLE]

Now, we are going to show that $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$. We argue by contradiction. Suppose there is a $x^{*}\in\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))$ such that $x^{*}\notin\operatorname{int}(\operatorname{dom}k)$. So $x^{*}$ has to be in $\operatorname{dom}k\setminus\operatorname{int}(\operatorname{dom}k)$, i.e., on the boundary of $\operatorname{int}(\operatorname{dom}k)$. Using the essential smoothness of $k$, by [38, Lemma 26.2] it follows that for any $y^{*}\in\operatorname{int}(\operatorname{dom}k)$, we have

[TABLE]

We fix an arbitrary $y^{*}\in\operatorname{int}(\operatorname{dom}k)$, and define the function $h:[0,1]\to\mathbb{R}$ as $h(\lambda)=L^{*}f^{*}(x^{*}+\lambda(y^{*}-x^{*}))-k(x^{*}+\lambda(y^{*}-x^{*}))$. Then relative smoothness implies that $h$ is a finite, continuous convex function on $[0,1]$. However, for such a function we must have $\limsup_{\lambda\downarrow 0}h_{+}^{\prime}(\lambda)<\infty$, since otherwise it could not be finite on $[0,1]$ by Lemma 2.9. By combining this with (33), it follows that

[TABLE]

Let $r:[0,1]\to\mathbb{R}$ be $r(\lambda)=f^{*}(x^{*}+\lambda(y^{*}-x^{*}))$, then this implies that $r^{\prime}_{+}(\lambda)\to\infty$ as $\lambda\downarrow 0$, and by (16) of Lemma 2.9, this contradicts the assumption that $f^{*}$ is finite in $\operatorname{dom}f^{*}$. Hence we must have $x^{*}\in\operatorname{int}(\operatorname{dom}k)$, and $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$. ∎

The next proposition gives a second-order sufficient condition for Assumption 3.1.3.

Proposition 3.7**.**

Let $f:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be essentially smooth, and $C^{2}$ on $\operatorname{int}(\operatorname{dom}f)$. Let $k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be Legendre convex, and $C^{2}$ on $\operatorname{int}(\operatorname{dom}k)$. If $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))\subseteq\operatorname{int}(\operatorname{dom}k)$, $\det\nabla\!\!\;^{2}k(x^{*})\neq 0$ for all $x^{*}\in\operatorname{int}(\operatorname{dom}k)$, and

[TABLE]

then $D_{k}(\nabla\!\!\;f(y),\nabla\!\!\;f(x))\leq L^{*}D_{f}(x,y)$ for every $x,y\in\operatorname{int}(\operatorname{dom}f)$.

Proof.

Let $x,y\in\operatorname{int}(\operatorname{dom}f)$, and let $W\subseteq\mathbb{R}^{d}$ be a bounded open neighborhood of the segment $I=[\nabla\!\!\;f(x),\nabla\!\!\;f(y)]=\{t\nabla\!\!\;f(x)+(1-t)\nabla\!\!\;f(y):0\leq t\leq 1\}$ such that $\mathrm{cl}(W)\subseteq\operatorname{int}(\operatorname{dom}k)$. Let $\delta>0$. Let $\epsilon>0$, and let $f_{\epsilon}:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$, $f_{\epsilon}(z)=f(z)+\epsilon\sqrt{1+\|z\|^{2}}$. Then $f_{\epsilon}$ is a Legendre convex function, and $\operatorname{dom}(f_{\epsilon})=\operatorname{dom}(f)$. We have $\nabla\!\!\;f_{\epsilon}(z)=\nabla\!\!\;f(z)+\epsilon(1+\|z\|^{2})^{-\frac{1}{2}}z$ and $\nabla\!\!\;^{2}f_{\epsilon}(z)=\nabla\!\!\;^{2}f(z)+\epsilon(1+\|z\|^{2})^{-\frac{3}{2}}((1+\|z\|^{2})I_{d}-z^{T}z)\succ 0$ for every $z\in\operatorname{int}(\operatorname{dom}f)$. So $\|\nabla\!\!\;f_{\epsilon}(z)-\nabla\!\!\;f(z)\|\leq\epsilon$ and $0\preceq\nabla\!\!\;^{2}f_{\epsilon}(z)-\nabla\!\!\;^{2}f(z)\preceq\epsilon I_{d}$ for every $z\in\operatorname{int}(\operatorname{dom}f)$. Let $I_{\epsilon}$ denote the segment $[\nabla\!\!\;f_{\epsilon}(x),\nabla\!\!\;f_{\epsilon}(y)]\subseteq\mathbb{R}^{d}$. Choose $\epsilon$ small enough so that

the $2\epsilon$-neighborhood of $I_{\epsilon}$ is in $W$ (so $\mathrm{dist}(I_{\epsilon},\mathbb{R}^{d}\setminus W)\geq 2\epsilon$). 2. 2.

$\epsilon I_{d}\preceq\frac{\delta}{2}[\nabla\!\!\;^{2}k(w)]^{-1}$ for every $w\in W$. 3. 3.

$\forall w_{1},w_{2}\in W$ s.t.  $\|w_{1}-w_{2}\|\leq\epsilon$, $[\nabla\!\!\;^{2}k(w_{1})]^{-1}\preceq(1+\frac{\delta}{2L^{*}})[\nabla\!\!\;^{2}k(w_{2})]^{-1}$ (uniform continuity of $(\nabla\!\!\;^{2}k)^{-1}$ on compact set $\mathrm{cl}(W)$ by Heine-Cantor theorem).

We will show that $(L^{*}+\delta)f_{\epsilon}^{*}-k$ is convex when restricted to the segment $I_{\epsilon}$. Let $w\in I_{\epsilon}$ and $z=(\nabla\!\!\;f_{\epsilon})^{-1}(w)\in\operatorname{int}(\operatorname{dom}f)$. Then $\nabla\!\!\;f_{\epsilon}(z)=w$, and since $\|\nabla\!\!\;f(z)-w\|\leq\epsilon$, we get $\nabla\!\!\;f(z)\in W$. We have $\nabla\!\!\;^{2}((L^{*}+\delta)f_{\epsilon}^{*}-k)(w)=(L^{*}+\delta)[\nabla\!\!\;^{2}f_{\epsilon}(z)]^{-1}-\nabla\!\!\;^{2}k(\nabla\!\!\;f_{\epsilon}(z))$, and we would like to show that this is $\succeq 0$. So we want to show $\nabla\!\!\;^{2}f_{\epsilon}(z)\preceq(L^{*}+\delta)[\nabla\!\!\;^{2}k(\nabla\!\!\;f_{\epsilon}(z))]^{-1}$. This follows from $\nabla\!\!\;^{2}f_{\epsilon}(z)\preceq\nabla\!\!\;^{2}f(z)+\epsilon I_{d}$ and $\epsilon I_{d}\preceq\frac{\delta}{2}[\nabla\!\!\;^{2}k(\nabla\!\!\;f_{\epsilon}(z))]^{-1}$ and $\nabla\!\!\;^{2}f(z)\preceq L^{*}[\nabla\!\!\;^{2}k(\nabla\!\!\;f(z))]^{-1}\preceq(L^{*}+\frac{\delta}{2})[\nabla\!\!\;^{2}k(\nabla\!\!\;f_{\epsilon}(z))]^{-1}$. So for small enough $\epsilon$’s $(L^{*}+\delta)f_{\epsilon}^{*}-k$ is indeed convex when restricted to $I_{\epsilon}$. Then $D_{k}(\nabla\!\!\;f_{\epsilon}(y),\nabla\!\!\;f_{\epsilon}(x))\leq(L^{*}+\delta)D_{f^{*}_{\epsilon}}(\nabla\!\!\;f_{\epsilon}(y),\nabla\!\!\;f_{\epsilon}(x))=(L^{*}+\delta)D_{f_{\epsilon}}(x,y)$, using the convexity of $(L^{*}+\delta)f_{\epsilon}^{*}-k$ on $I_{\epsilon}$ combined with the same limiting argument as in (17), and Lemma 3.2. Taking $\epsilon\downarrow 0$, and then $\delta\downarrow 0$ we get $D_{k}(\nabla\!\!\;f(y),\nabla\!\!\;f(x))\leq L^{*}D_{f}(x,y)$. ∎

3.4 Convergence rates for dual space preconditioned gradient descent

In this section we show that Assumption 3.1 is sufficient to provide convergence rates for Algorithm 1.1 on essentially smooth convex $f$. We find that $k(\nabla\!\!\;f(x_{i})))-k(0)$ converges with rate $\mathcal{O}(i^{-1})$. Under an additional dual relative strong convexity condition, we find that $f(x_{i})-f(x_{\min})$ converges with rate $\mathcal{O}((1-\mu^{*}/L^{*})^{i})$. We begin with the following descent lemma.

Lemma 3.8** (Descent lemma).**

Let $f,k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ satisfy Assumption 3.1. If $x_{0}\in\operatorname{int}(\operatorname{dom}f)$, then for all $i>0$, the iterates $x_{i}$ of Algorithm 1.1 are such that $x_{i}\in\operatorname{int}(\operatorname{dom}f)$ and for all $x\in\operatorname{int}(\operatorname{dom}f)$,

[TABLE]

Proof.

Let $C=\operatorname{int}(\operatorname{dom}f)$. We proceed by induction. For $i=0$ we have $x_{0}\in C$ by assumption. Now, for $i>0$, assume the induction hypothesis for $x_{i-1}$. Define,

[TABLE]

for $\lambda>0$. Because $x_{i-1}\in\operatorname{int}(\operatorname{dom}f)\neq\emptyset$, the set $S=\{\lambda\geq L^{*}:x_{\lambda}\in C\}$ is not empty. Let $x\in C$. Let $x^{*}=\nabla\!\!\;f(x)$, $x^{*}_{i-1}=\nabla\!\!\;f(x_{i-1})$ and $x^{*}_{\lambda}=\nabla\!\!\;f(x_{\lambda})$ for $\lambda\in S$. The following identities follow by our definition of $x_{\lambda}$ and some algebra.

[TABLE]

Combining (37) and (38), we get

[TABLE]

Putting everything together, we have

[TABLE]

$(a)$ follows from $L^{*}$-smoothness, $(b)$ from $L^{*}\leq\lambda$ and the non-negativity of the Bregman divergence, $(c)$ from (39), and $(d)$ by definition and simple algebra. Taking $x=x_{i-1}$ and recalling the definition of $x^{*}_{i-1}$ and $x^{*}_{\lambda}$ reveals that

[TABLE]

Now, our goal is to show that $x_{i}=x_{L^{*}}\in\operatorname{int}(\operatorname{dom}f)$ by showing that $L^{*}\in S$. We proceed by contradiction, so suppose $L^{*}\notin S$. Then $x_{L^{*}}\in\mathbb{R}^{d}\setminus\operatorname{int}(\operatorname{dom}f)$. Hence we can find $\Lambda\geq L^{*}$ such that $x_{\Lambda}\in\partial(\operatorname{dom}f)$. Now take a sequence $\lambda_{j}\to\Lambda$ such that $\lambda_{j}>\Lambda$. By the above discussion for all $j\geq 0$ we have $k(\nabla\!\!\;f(x_{\lambda_{j}}))\leq k(\nabla\!\!\;f(x_{i-1}))$. $k$ being minimized at [math] means it satisfied Lemma 2.3 and thus is radially unbounded. This implies that $\left\lVert\nabla\!\!\;f(x_{\lambda_{j}})\right\rVert\leq c$ for some $c>0$ and all $j\geq 0$. But this contradicts the requirement that $\left\lVert\nabla\!\!\;f(x_{\lambda_{j}})\right\rVert\to\infty$ since $x_{\lambda_{j}}\to x_{\Lambda}\in\partial(\operatorname{dom}f)$ from the assumption of essential smoothness. This completes the proof that $x_{i}=x_{L^{*}}\in\operatorname{int}(\operatorname{dom}f)$. Since $L^{*}\in S$, (40) ensures that (35) holds. ∎

We can now provide convergence rates for our method.

Theorem 3.9**.**

Let $f,k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ satisfy Assumption 3.1. If $x_{0}\in\operatorname{int}(\operatorname{dom}f)$, then for all $i>0$ the iterates of Algorithm 1.1 satisfy

[TABLE]

In particular, $\nabla\!\!\;f(x_{i})\to 0$. If additionally $f$ is Legendre convex and there exists $\mu^{*}>0$ such that $\mu^{*}D_{f}(x,y)\leq D_{k}(\nabla\!\!\;f(y),\nabla\!\!\;f(x))$ for all $x,y\in\operatorname{int}(\operatorname{dom}f)$, then for all $i>0$ the iterates of Algorithm 1.1 satisfy

[TABLE]

Proof of Theorem 3.9.

Let $C=\operatorname{int}(\operatorname{dom}f)$. We have $x_{i}\in C$ and $k(\nabla\!\!\;f(x_{i}))\leq k(\nabla\!\!\;f(x_{i-1}))$ by the Descent Lemma 3.8. We also have $x_{\min}\in C$ by Lemma 2.2. Finally, (35) of Lemma 3.8 with $x=x_{\min}$ gives us,

[TABLE]

Putting this together, we get

[TABLE]

Dividing by $i$ gives our first result. This implies that $k(\nabla\!\!\;f(x_{i}))\to k(0)$, which implies that $\nabla\!\!\;f(x_{i})\to 0$ by continuity and the uniqueness of $k$’s minimum. Now, assume that $f$ is Legendre convex and there exists $\mu^{*}>0$ such that $\mu^{*}D_{f}(x,y)\leq D_{k}(\nabla\!\!\;f(y),\nabla\!\!\;f(x))$ for all $x,y\in\operatorname{int}(\operatorname{dom}f)$. For all $i>0$,

[TABLE]

where $(a)$ follows from (44) and the non-negativity of $k(x^{*})-k(0)$. $(b)$ follows from dual relative strong convexity. This inequality implies our desired result. ∎

Theorem 3.9 guarantees the convergence of the iterates of Algorithm 1.1 under the assumption that dual relative smoothness hold globally for a fixed $L^{*}$. Unfortunately it may be difficult to to derive a tight bound on $L^{*}$ or small $L^{*}$ may be appropriate locally. In this case, it may be useful to use a line search to choose $L^{*}$. Consider the following generalization of the update rule of Algorithm 1.1,

[TABLE]

where $L^{*}_{i}>0$ is allowed to depend on the iteration. The next proposition shows that, under suitable assumptions, (47) converges with rates analogous to Theorem 3.9.

Proposition 3.10** (Adaptive step sizes).**

Let $f:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be a proper closed convex function that is differentiable on $\operatorname{int}(\operatorname{dom}f)\neq\emptyset$ and minimized at $x_{\min}$. Let $k:\mathbb{R}^{d}\to\mathbb{R}\cup\{\infty\}$ be a proper closed convex function that is differentiable on $\nabla\!\!\;f(\operatorname{int}(\operatorname{dom}f))$. If $x_{0}\in\operatorname{int}(\operatorname{dom}f)$ and for all $i>0$ the iterates $x_{i}$ in (47) satisfy

$x_{i}\in\operatorname{int}(\operatorname{dom}f)$, 2. 2.

$k(\nabla\!\!\;f(x_{i}))\leq k(\nabla\!\!\;f(x_{i-1}))$, 3. 3.

$k(\nabla\!\!\;f(x_{i}))-k(0)\leq L^{*}_{i-1}(f(x_{i-1})-f(x_{i}))$,

then we have

[TABLE]

Remark 3.11*.*

In practice, a possible choice of step sizes is

[TABLE]

If $L^{*}$ is the smallest real number such that $f$ is dual $L^{*}$-smooth relative to $k$ (see Lemma 3.6 for an equivalent condition), then this scheme satisfies that $L^{*}_{i-1}<2L^{*}$ for every $i>0$ (hence we are making steps that are almost as large or larger as if we would use the smallest possible fixed $L^{*}$, without knowing the value of $L^{*}$ in advance). The search through the set in (49) for finding $L^{*}_{i}$ can be initialized at $L^{*}_{i-1}$.

Proof of Proposition 3.10.

The proof follows similar lines as in the previous case. First, by summing up the inequalities from 3, we obtain that

[TABLE]

and using 2., it follows that $\sum_{1\leq j\leq i}[k(\nabla\!\!\;f(x_{j}))-k(0)]\geq i(k(\nabla\!\!\;f(x_{i}))-k(0))$. The result follows directly. ∎

An important question that we do not address in this section is whether the sub-linear convergence of $k(\nabla\!\!\;f(x_{i}))-k(0)$ implies specific rates of convergence of other quantities of interest. These might be, for example, $\left\lVert x_{i}-x_{\min}\right\rVert$ or $f(x_{i})-f(x_{\min})$. Rates for these will likely depend on both $f$ and $k$.

4 Applications

4.1 Exponential Penalty Functions

Consider the following problem.

[TABLE]

where $c\in\mathbb{R}^{d}$, $b\in\mathbb{R}^{n}$, and $A\in\mathbb{R}^{n\times d}$. Associate with this linear program the following relaxation into an unconstrained problem: $\min_{x\in\mathbb{R}^{d}}f_{\tau}(x)$ for

[TABLE]

where $\tau>0$ and $A_{i}$ is the $i$th row of $A$ (a row vector). This approximation of (LP) with exponential penalty functions was studied by several authors (see [44, 21, 37, 5]) and is directly useful in the machine learning literature for boosting (see, e.g., [32]). Here we design a dual reference function for $f_{\tau}$ under the following assumptions.

Assumption 4.1**.**

Suppose that the following hold for problem (LP).

$\left\lVert A_{i}\right\rVert=1$* for $1\leq i\leq n$.* 2. 2.

$A\in\mathbb{R}^{n\times d}$* is of full rank $d\leq n$.* 3. 3.

$P=\{x\in R^{n}:Ax\leq b\}$* is a polytope, which is contained in a Euclidean ball of radius $R>0$ and contains a Euclidean ball of radius $r>0$.*

The dual reference function will be designed so that is smooth relative to $f_{\tau}^{*}$ and Algorithm 1.1, with appropriate step-size choices, converges with global guarantees.

Define the dual reference function $k:\mathbb{R}^{d}\to\mathbb{R}$,

[TABLE]

This behaves like a quadratic $\|x^{*}\|^{2}/2$ near its minimum $x^{*}=0$ and like $\|x^{*}\|$, i.e., grows linearly, at infinity. It is also possible to verify that $k$ is Legendre convex. Furthermore, we have:

[TABLE]

Hence, $[\nabla^{2}\!\!\;k(x^{*})]^{-1}\succeq(1+\left\lVert x^{*}\right\rVert)I$. From Proposition 3.4 and this inequality it follows that the fact that $k$ is $L^{*}$-smooth relative to $f^{*}_{\tau}$ is implied by

[TABLE]

This is the strategy of the following theorem, which shows that $f_{\tau}$ is dual smooth to this choice of $k$ under our assumptions.

Proposition 4.2**.**

Under Assumption 4.1 for $f_{\tau}$ defined in (50) and $k$ defined in (51), we have that

[TABLE]

where the dual relative smoothness constant is given by

[TABLE]

Here, $\left\lVert A^{T}A\right\rVert$ is the induced matrix norm, and

[TABLE]

Because $f_{\tau}$ and $k$ are Legendre convex, $k$ is smooth relative to $f_{\tau}^{*}$ and Theorem 3.9 implies that Algorithm 1.1 converges with $k(\nabla\!\!\;f(x_{i}))$ converging at a rate $\mathcal{O}(1/i)$.

Remark 4.3*.*

From Theorem 3.9, we have

[TABLE]

This suggests that, if we can start from an initial point within the polytope, then we can reach a point where $\|\nabla\!\!\;f_{\tau}(x)\|$ is significantly less than $\|c\|$ (which is expected to be near the minimum) in polynomial amount of steps, depending on the conditioning $R/r$ and the value of $\tau$. The step-size $1/L^{*}_{i}$ can also be chosen adaptively, as explained in Proposition 3.10. Near the minimum, both $f_{\tau}(x)$ and $k(x^{*})$ behave like quadratic functions, so local linear convergence rates hold. We believe that this iterative scheme is reasonably efficient for high dimensional well-conditioned polytopes, but in other less well conditioned instances it is outperformed by existing algorithms such as multiplicative weights [4] or [20], which is based on Newton’s method (hence uses second-order information).

Proof of Proposition 4.2.

Note that $1\leq\eta\leq n$, because $\left\lVert A_{i}\right\rVert=1$. Let $\alpha(x):=\max_{i\in[n]}(A_{i}x-b_{i})$. Then $\alpha(x)<0$ inside the polytope and $\alpha(x)>0$ outside of it. By differentiation, we have

[TABLE]

Note that $f_{\tau}$ is defined everywhere and differentiable. Furthermore, under our assumption that $\operatorname{rank}(A^{T}A)=\operatorname{rank}(A)=d$, it is evidently strictly convex and therefore Legendre.

The Hessian of $f_{\tau}$ satisfies

[TABLE]

Because $\eta\geq 1$, it is clear that the claim of the theorem holds for every $x$ where $\alpha(x)\leq 0$ (i.e. inside the polytope or on its boundary). From now on we will assume that $x$ is such that $\alpha(x)>0$ (outside of the polytope). Let $x_{c}$ be a minimizer of $\alpha(x)$ (at least one exists since the polytope is compact and $\alpha(x)$ is a continuous function), then using the assumption $\|A_{i}\|=1$ it follows that $\alpha(x_{c})=-r<0$. Hence $x\neq x_{c}$. We are going to need an upper bound on $\left\lVert x-x_{c}\right\rVert$, which we will obtain as follows. By the definitions, we have $A_{i}x_{c}\leq-r+b_{i}$ and $A_{i}x=A_{i}x-b_{i}+b_{i}\leq\alpha(x)+b_{i}$, hence

[TABLE]

Therefore $x_{c}+\frac{r}{\alpha(x)+r}(x-x_{c})\in P\subset B_{x_{c}}(2R)$, so

[TABLE]

Let $\mathcal{I}=\{i\in[n];\,A_{i}x-b_{i}>0\}$, $\mathcal{J}=\{i\in[n];\,A_{i}x-b_{i}\leq 0\}$, and

[TABLE]

Then $\nabla\!\!\;f_{\tau}(x)=G_{\mathcal{I}}(x)+G_{\mathcal{J}}(x)+c$. We have

[TABLE]

Here, $(a)$ follows from the facts that there is a $j\in\mathcal{I}$ such that $A_{j}(x-x_{c})=\alpha(x)+b_{j}-A_{j}x_{c}\geq\alpha(x)+r$ and the fact that $A_{i}(x-x_{c})\geq b_{i}+r-b_{i}>0$ holds for every $i\in\mathcal{I}$. $(b)$ follows from (61). From (60) we obtain that

[TABLE]

Hence (53) follows from the facts that $\|G_{\mathcal{J}}(x)\|\leq\eta$ and $\eta+\left\lVert c\right\rVert\geq 1$. As discussed (54) follows from $[\nabla^{2}\!\!\;k(x^{*})]^{-1}\succeq(1+\left\lVert x^{*}\right\rVert)I$. ∎

4.2 $p$-norm Regression

Consider the following $p$-norm regression problem,

[TABLE]

where $A\in\mathbb{R}^{n\times d}$, $d\ll n$, $b\in\mathbb{R}^{n}$, and $p\geq 1$. This problem is a useful abstraction for some important graph problems, including Lipschitz learning on graphs [29] and $\ell_{p}$-norm minimizing flows [1]. Algorithms specialized for $p$-norm regression have recently been studied in the theoretical computer science literature by several authors (see, e.g., [17, 2] and references therein). In this subsection, we design an appropriate dual reference function for (pnorm) under the following assumptions. Let $A_{i}$ denote the rows of $A$ (as row vectors).

Assumption 4.4**.**

Suppose that the following hold for problem (pnorm).

$2\leq p<\infty$. 2. 2.

$A$* is full rank $d$, and for all $x\in\mathbb{R}^{d}$ there is a subset $I(x)\subset[n]$ such that $A_{i}x\neq b_{i}$ for all $i\in I(x)$, and $\mathrm{span}\{A_{i}:i\in I(x)\}=\mathbb{R}^{d}$.* 3. 3.

$c_{G}=\inf_{\|s\|=1}\left\lVert As\right\rVert_{p}^{p}>0$. 4. 4.

$c_{H}=\inf_{u,v\in\mathbb{R}^{d}:\|u\|=1,\|v\|=1}\sum_{i=1}^{n}\left\lvert A_{i}u\right\rvert^{p-2}(A_{i}v)^{2}>0$.

Remark 4.5*.*

Although these assumptions seem restrictive, we can show that, if $n\geq 2d-1$ and $(A_{i})_{1\leq i\leq n}$ and $(b_{i})_{1\leq i\leq n}$ are chosen as independent random variables with densities that are absolutely continuous with respect to the Lebesgue measure on $\mathbb{R}^{d}$ and $\mathbb{R}$, then the assumptions hold with probability 1. Assumption 2 is implied by the stronger assumption that any $d$ rows of $A$ define a full rank $d$ matrix, and the maximal number of equalities $A_{i}x=b_{i}$ that hold for any $x$ is no more than $d$. This stronger version of Assumption 2, and Assumption 3 holds with probability 1 under the random allocation due to the fact that the set of real valued $d\times d$ matrices with determinant [math] has Lebesgue-measure [math] in $\mathbb{R}^{d\times d}$ (due to the fact that the determinant is a multivariate polynomial of the entries, and the zero set of such polynomials has Lebesgue measure zero unless they are constant 0, see [19]). The minimum in Assumption 4 is achieved for some $u_{\min}$ and $v_{\min}$ due to continuity and compactness of the unit sphere. Since any $d$ rows of $A$ form an independent basis with probability 1, it follows that $u$ and $v$ can be orthogonal to at most $d-1$ of them, respectively, so using $n\geq 2d-1$ there exists an $i$ in the sum $\sum_{i=1}^{n}\left\lvert A_{i}u_{\min}\right\rvert^{p-2}(A_{i}v_{\min})^{2}$ that is non-zero, hence Assumption 4 holds.

Consider the dual reference function $k:\mathbb{R}^{d}\to\mathbb{R}$,

[TABLE]

for $q=\frac{p}{p-1}$ (hence $\tfrac{1}{p}+\tfrac{1}{q}=1$). This behaves like a quadratic $\|x^{*}\|^{2}/2$ near its minimum $x^{*}=0$ and like $\|x^{*}\|^{q}/q$ at infinity. For this $k$, we have

[TABLE]

As the next theorem shows dual relative strong convexity and smoothness of $k$ relative to the conjugate of (pnorm) hold under our assumptions.

Proposition 4.6**.**

Let $f(x)=\left\lVert Ax-b\right\rVert_{p}^{p}$ be the $p$-norm objective. Under Assumption 4.4 for $k$ defined in (64), there exists $\mu^{*},L^{*}>0$ such that

[TABLE]

See (76) and (77) for the definitions of $\mu^{*}$ and $L^{*}$. Because $f$ and $k$ are Legendre convex, $k$ is smooth and strongly convex relative to $f^{*}$ and Theorem 3.9 implies that Algorithm 1.1 converges with $f(x_{i})-f(x_{\min})$ converging at a linear rate $\mathcal{O}((1-\mu^{*}/L^{*})^{i})$.

To test the empirical performance of this method, we have implemented it with $A_{i}$, $b$, and $x_{0}$ i.i.d. as standard normals for power $p=4$, $d\in\{10^{2},10^{3},10^{4}\}$, and $n=10d$. The inverse step-size $L^{*}_{0}$ was chosen to be $L^{*}_{0}=1$ initially, and multiplied by 2 if the function value would increase due to too large steps (hence this was chosen adaptively in the beginning, but $L^{*}_{i}$ was never decreased later on). As Figure 1 shows, empirically our method seems to be performing well, with high precision achieved after 50-80 gradient evaluations, and the convergence rate seems to be mostly unaffected by the dimension $d$. Hence in this random setting dual space preconditioning is indeed very efficient, and competitive with previous works [17, 2, 1] which had dimension dependent convergence rates. We think that based on Proposition 4.6, it can be shown that with high probability, dimension-free convergence rates hold in this random scenario when the number of vectors $n$ tends to infinity (the proof would be based on concentration inequalities for empirical processes, see e.g. [14] for an overview of such inequalities). Note however that we do not believe this always to be the case for general non-random $A$ and $b$, and there could be instances of very poor conditioning (such as when $n\approx d$) where the homotopy method of [17] or the IRLS method of [3] could perform better.

The proof of Proposition 4.6 is based on the following two lemmas.

Lemma 4.7** (Bounds on the gradient).**

Let $f(x)=\left\lVert Ax-b\right\rVert_{p}^{p}$ be the $p$-norm objective for (pnorm). Under Assumption 4.4, we have

[TABLE]

for all $x\in\mathbb{R}^{d}$, with constants

[TABLE]

Proof.

By differentiation, we have

[TABLE]

thus

[TABLE]

and the lower bound follows from Assumption 4.4 by straightforward rearrangement. For the upper bound, notice that

[TABLE]

hence the result follows. The last step uses Fenchel-Young, and rearrangement. ∎

Lemma 4.8** (Bounds on the Hessian).**

Let $f(x)=\left\lVert Ax-b\right\rVert_{p}^{p}$ be the $p$-norm objective. Suppose that Assumption 4.4 holds, and let

[TABLE]

Then $\rho_{H}>0$, and we have

[TABLE]

for all $x\in\mathbb{R}^{d}$, with constants

[TABLE]

Proof.

We have by differentiation

[TABLE]

Notice that using the fact that $|a-b|^{p-2}+|b|^{p-2}\geq 2^{-(p-1)}|a|^{p-2}$, we have

[TABLE]

Let $R_{H}$ be as in (69), then using the above bound, we can see that for $\|x\|\geq R_{H}$, we have

[TABLE]

Since the minimum of the continuous function $\lambda_{\min}(\nabla^{2}\!\!\;f(x))$ is achieved on the compact set $B_{R_{H}}$, and by the second part of Assumption 4.4, it cannot be zero, and hence $\rho_{H}>0$ and $\nabla\!\!\;^{2}f(x)\succeq\rho_{H}I$ for every $x\in B_{R_{H}}$. The lower bound in (71) follows by combining this with (73). For the upper bound, using the inequality $|a+b|^{p-2}\leq 2^{p-3}(|a|^{p-2}+|b|^{p-2})$, we obtain that

[TABLE]

∎

Now we are ready to prove our main result in this section.

Proof of Proposition 4.6.

First, both $f$ and $k$ are Legendre convex in this case. This is easy to verify for $k$, and evidently $f$ is differentiable everywhere. To verify strict convexity of $f$, note that $\nabla^{2}\!\!\;f(x)\succ 0$ under part two of Assumption 4.4. Since both $f$ and $k$ are twice differentiable, by Proposition 3.4, it suffices to check that (66) holds for the linear convergence of Algorithm 1.1. We have by differentiation,

[TABLE]

Now it is easy to see that for $p\in[2,\infty)$, we have $q=p/(p-1)\in(1,2]$ and it is not difficult to verify that $\nabla\!\!\;^{2}k$ satisfies that for all $x^{*}\in\mathbb{R}^{d}$,

[TABLE]

The claim of the theorem now follows by some straightforward rearrangement using Lemmas 4.7 and 4.8, with constants

[TABLE]

∎

5 Discussion

In this paper we introduced a non-linear preconditioning scheme for gradient descent on Legendre convex functions $f$ that converges under generalizations of the standard Lipschitz assumption on $\nabla\!\!\;f$. There are at least two interpretations of this method. The first is as a generalization of gradient descent in which the update direction is preconditioned by the gradient map $\nabla\!\!\;k$ of a designed dual reference, Legendre convex function $k$. The second interpretation is as a Bregman gradient method in the dual space, which minimizes the designed $k$ while the conjugate $f^{*}$ plays the role of the “reference function”. The choice of $k$ affects the conditioning of our method, which is made explicit in our analysis through a relative smoothness condition between $k$ and $f^{*}$. The dual relative conditions admit non-smooth $f$ and $k$, and are provably distinct dual cousins of the relative smoothness conditions introduced by [7]. $k$ serves as a model of the convex conjugates $f^{*}$ in a certain problem class. In section 4, we show how this method can be applied to exponential penalty functions (see, e.g., [21, 20]) and $p$-norm regression (see [17, 2] and references therein) with global convergence rate guarantees.

Algorithm 1.1 is related to a number of existing methods, some of which are subject to the analysis we provide. The most notable of these is the method of steepest descent with respect to a given norm $\left\lVert\cdot\right\rVert$ (now not necessarily Euclidean). Here we follow the exposition of Boyd and Vandenberghe [15, sect. 4.9]. The steepest descent iteration is given by

[TABLE]

and $\left\lVert x^{*}\right\rVert_{*}=\sup_{\left\lVert x\right\rVert\leq 1}\left\langle x,x^{*}\right\rangle$ is the dual norm of $\left\lVert\cdot\right\rVert$. The identity $\partial(\left\lVert x^{*}\right\rVert_{*}^{2}/2)=\left\lVert x^{*}\right\rVert_{*}\arg\max\{\left\langle x^{*},x\right\rangle:\left\lVert x\right\rVert\leq 1\}$ for all $x^{*}\in\mathbb{R}^{d}$ implies that for strictly convex and differentiable $\left\lVert\cdot\right\rVert_{*}$, the steepest descent method (78) is a special case of dual preconditioned gradient descent with $k(x^{*})=\left\lVert x^{*}\right\rVert_{*}^{2}/2$. Our analysis does not apply in the case of other norms or normalized steepest descent [15]. Algorithm 1.1 also generalizes the rescaled gradient method of [47, sect. 2.2]. Thus, our method may be seen as a generalization of the steepest descent method and rescalings of gradient descent. Dual preconditioning is more distantly related to the dual gradient methods [43, 9]. These methods are designed for problems with non-smooth, but strongly convex structure. They exploit the duality between classical smoothness and strong convexity by applying smooth minimization algorithms to a dual problem. Similarly, Algorithm 1.1 can be seen as a move to the dual space, in which a dual problem $k(x^{*})\approx f^{*}(x^{*})-\left\langle x^{*},x_{\min}\right\rangle$ (dual to $f(x)+\delta_{x=x_{\min}}(x)$) is minimized by a Bregman gradient method. Thus, dual gradient methods and dual preconditioning are most easily applied when the dual structure is relatively more benign to model than the primal structure, e.g., when $f$ has super-quadratic growth.

There are a couple natural questions that arise from this work. First, it may be useful to pursue the analogy with dual gradient methods further and to design methods for the general composite model that exploit dual relative smoothness. Second, there is still considerable difficulty in the design of $k$. Thus, it may be productive to investigate whether methods from linear preconditioning (see [10] for a review), such as incomplete factorizations or sparse approximate inverses, can be generalized to the non-linear setting for the design of $k$. Nonetheless, the dual relative conditions studied in this work provide new avenues for improving the conditioning of optimizers via hard-won domain-specific knowledge.

Acknowledgements

We thank the anonymous referees for their insightful comments that helped us to improve the paper. We thank David Balduzzi for insightful comments, Patrick Rebeschini for suggesting exponential penalty functions and Sushant Sachdeva for suggesting $p$-norm regression. CJM acknowledges the support of the Institute for Advanced Study and the James D. Wolfensohn Fund, the support of a DeepMind Graduate Scholarship, and the support of the Natural Sciences and Engineering Research Council of Canada under reference number PGSD3-460176-2014. YWT’s research leading to these results has received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013) ERC grant agreement no. 617071. This material is based upon work supported in part by the U.S. Army Research Laboratory and the U. S. Army Research Office, and by the U.K. Ministry of Defence (MoD) and the U.K. Engineering and Physical Research Council (EPSRC) under grant number EP/R013616/1.