Distribution-Dependent Analysis of Gibbs-ERM Principle

TL;DR

This paper provides distribution-dependent bounds on the excess risk of Gibbs-ERM learners, linking it to the effective dimension of the problem and extending understanding of learning in large hypothesis spaces.

Contribution

It introduces the first analysis connecting Gibbs-ERM excess risk bounds to the effective dimension, improving upon previous distribution-agnostic results.

Findings

Excess risk is controlled by the effective dimension of the Hessian at local minima.

Distribution-dependent bounds can be tighter than previous results.

Analysis extends from local minima to global excess risk bounds.

Abstract

Gibbs-ERM learning is a natural idealized model of learning with stochastic optimization algorithms (such as Stochastic Gradient Langevin Dynamics and ---to some extent--- Stochastic Gradient Descent), while it also arises in other contexts, including PAC-Bayesian theory, and sampling mechanisms. In this work we study the excess risk suffered by a Gibbs-ERM learner that uses non-convex, regularized empirical risk with the goal to understand the interplay between the data-generating distribution and learning in large hypothesis spaces. Our main results are distribution-dependent upper bounds on several notions of excess risk. We show that, in all cases, the distribution-dependent excess risk is essentially controlled by the effective dimension $\mathrm{tr}\left(\boldsymbol{H}^{\star} (\boldsymbol{H}^{\star} + \lambda \boldsymbol{I})^{-1}\right)$ of the problem, where $\boldsymbol{H}^{\star}$ is the Hessian matrix of the risk at a local minimum. This is a well-established notion of effective dimension appearing in several previous works, including the analyses of SGD and ridge regression, but ours is the first work that brings this dimension to the analysis of learning using Gibbs densities. The distribution-dependent view we advocate here improves upon earlier results of Raginsky et al. (2017), and can yield much tighter bounds depending on the interplay between the data-generating distribution and the loss function. The first part of our analysis focuses on the localized excess risk in the vicinity of a fixed local minimizer. This result is then extended to bounds on the global excess risk, by characterizing probabilities of local minima (and their complement) under Gibbs densities, a results which might be of independent interest.

Equations

$$\underset{\mathbf{w}\in {\mathbb{R}}^d}{\mathrm{argmin}}\left\{{\hat{R}}_S(\mathbf{w})+\lambda \Vert \mathbf{w}{\Vert }^2\right\},\lambda >0,$$

$${\hat{R}}_S(\mathbf{w})=\frac{1}{m}\sum _{i=1}^m\ell (\mathbf{w},z_i).$$

$${\hat{p}}_{S,\gamma }(\mathbf{w})=\frac{1}{Z}{e}^{-\gamma \left({\hat{R}}_S(\mathbf{w})+\lambda \Vert \mathbf{w}{\Vert }^2\right)},\gamma >0,$$

$${\hat{\mathbf{w}}}_{t+1}={\hat{\mathbf{w}}}_t-\eta \nabla {\hat{R}}_{S,\lambda }({\hat{\mathbf{w}}}_t)+\sqrt{\frac{2\eta }{\gamma }}{\mathbf{\xi }}_{t}t=0,1,2,\dots$$

$$\mathrm{d}\mathbf{w}(t)=-\nabla {\hat{R}}_{S,\lambda }(\mathbf{w}(t))\mathrm{d}t+\sqrt{\frac{2}{\gamma }}\mathrm{d}\mathbf{b}(t),t\ge 0$$

$$\underset{\hat{p}\in {\mathcal{M}}_1}{\mathrm{arginf}}\left\{\underset{\mathbf{w}\sim \hat{p}}{E}\left[{\hat{R}}_S(\mathbf{w})\right]+\frac{1}{\gamma }\mathrm{KL}\left(\hat{p}||\mathcal{N}(0,{\lambda }^{-1}\mathbf{I})\right)\right\},$$

$$\underset{S,A}{E}\left[R(A(S))\right]-R({\mathbf{w}}^{\star })=\underset{\text{Computational excess risk}}{\underbrace{\underset{S,A,\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}\left[R(A(S))-R(\mathbf{w})\right]}}+\underset{\text{Statistical excess risk}}{\underbrace{\underset{\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}\left[R(\mathbf{w})\right]-R({\mathbf{w}}^{\star })}}.$$

$$\frac{(\gamma +d{)}^2}{{\lambda }_{\star }m}+\frac{d}{\gamma }$$

$$\frac{1}{{\lambda }_{\star }}=\tilde{\mathcal{O}}\left(\frac{1}{\gamma (d+\gamma )}\right)+\left(1+\frac{d}{\gamma }\right)e^{\tilde{\mathcal{O}}(\gamma +d)}.$$

$${\mathbf{w}}_{\lambda }^{\star }\in \underset{\mathbf{w}\in {\mathbb{R}}^d}{\mathrm{argmin}}\left\{R(\mathbf{w})+\lambda \Vert \mathbf{w}{\Vert }^2\right\}.$$

$$\Delta ({\mathbf{w}}_{\lambda }^{\star })=\underset{S}{E}\left[\underset{\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}\left[R(\mathbf{w})-R({\mathbf{w}}_{\lambda }^{\star })\mid \mathbf{w}\in {\mathcal{E}}^{\star }\left(r\right)\right]\right],$$

$$\Delta ({\mathbf{w}}_{\lambda }^{\star })\lesssim \frac{1}{\gamma }\mathrm{tr}\left({\mathbf{H}}^{\star }{\mathbf{H}}_{\lambda }^{\star -1}\right)+\epsilon (r)+\sqrt{\gamma \epsilon (r)}+\frac{\gamma }{\sqrt{m}}$$

$$\Delta ({\mathbf{w}}^{\star })\lesssim \frac{1}{\gamma }\mathrm{rank}({\mathbf{H}}^{\star })+\text{poly}(m,\gamma ,r,{\lambda }_{\min }({\mathbf{H}}^{\star }))$$

$$\Delta ({\mathbf{w}}_{\lambda }^{\star })\lesssim \frac{1}{\gamma }\mathrm{tr}\left({\mathbf{H}}^{\star }{\mathbf{H}}_{\lambda }^{\star -1}\right)+\frac{\gamma }{\sqrt{m}}.$$

$$\Delta (\pi )=\underset{S}{E}\left[\underset{\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}\left[R(\mathbf{w})\right]\right]-\underset{I\sim \pi }{E}\left[R({\mathbf{w}}_{\lambda ,I}^{\star })\right].$$

$${\pi }_{\gamma ,r}(i)=\frac{{\mathbb{P}}_{\gamma }({\mathcal{E}}_i^{\star }(r))}{{\sum }_{j\in \mathcal{I}}{\mathbb{P}}_{\gamma }({\mathcal{E}}_j^{\star }(r))},i\in \mathcal{I},$$

$$\Delta ({\pi }_{\gamma ,r})\lesssim \frac{1}{\gamma }E\left[\mathrm{tr}\left({\mathbf{H}}_I^{\star }{\mathbf{H}}_{\lambda ,I}^{\star -1}\right)\right]+\frac{\gamma }{\sqrt{m}}+E[{\epsilon }_I(r)]+\sqrt{\gamma E[{\epsilon }_I(r)]}+{\mathbb{P}}_{\gamma }({\mathcal{C}}^{\star }(r)),$$

$${\mathbb{P}}_{\gamma }({\mathcal{C}}^{\star }(r))\le 1-\left(1-d{e}^{-r^2\gamma {\alpha }_{d/2}}\right)\sum _{i\in \mathcal{I}}{e}^{-\frac{1}{3}\gamma {\epsilon }_i(r)},$$

$${\pi }_{\infty }(i)=\frac{1}{{\sum }_{j\in {\mathcal{I}}^{\text{glob}}}\sqrt{\frac{\det ({\mathbf{H}}_{\lambda ,i}^{\star })}{\det ({\mathbf{H}}_{\lambda ,j}^{\star })}}},i\in {\mathcal{I}}^{\text{glob}}$$

$${\Delta }_r^{\infty }=\underset{I\sim {\pi }_{\infty }}{E}\left[\underset{S}{E}\left[\underset{\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}\left[R(\mathbf{w})\mid \mathbf{w}\in {\mathcal{E}}_I^{\star }(r)\right]\right]-R({\mathbf{w}}_{\lambda ,I}^{\star })\right],r>0,$$

$${\Delta }_r^{\infty }\lesssim \frac{1}{\gamma }E\left[\mathrm{tr}\left({\mathbf{H}}_I^{\star }{\mathbf{H}}_{\lambda ,I}^{\star -1}\right)\right]+E\left[{\epsilon }_I(r)\right]+\sqrt{\gamma E\left[{\epsilon }_I(r)\right]}+\frac{\gamma }{\sqrt{m}}.$$

$$\mathrm{KL}(p,q)$$

$$E\left[R(\mathbf{w})-R({\mathbf{w}}_{\lambda }^{\star })\mid \mathbf{w}\in {\mathcal{E}}^{\star }(r)\right]$$

$$+E\left[{\hat{R}}_S(\mathbf{w})-R({\mathbf{w}}_{\lambda }^{\star })\mid \mathbf{w}\in {\mathcal{E}}^{\star }(r)\right]$$

$${\hat{p}}_{S,\gamma }(\mathbf{w})\propto e^{-\frac{\gamma }{m}{\sum }_{i=1}^{m}f(\mathbf{w},z_i)}\gamma >0$$

$$\operatorname*{\mathbb{E}}_{S}\left[\operatorname*{\mathbb{E}}_{\boldsymbol{w}\sim\widehat{p}_{S,\gamma}}\left[R(\boldsymbol{w})-\widehat{R}_{S}(\boldsymbol{w})\,\Big{|}\,\boldsymbol{w}\in A\right]\right]\leq\frac{4\sigma^{2}\gamma}{m}~{}.$$

$${\hat{R}}_S(\mathbf{w})-R({\mathbf{w}}_{\lambda }^{\star })=\tilde{\mathcal{O}}\left(\frac{d}{\gamma }\right)$$

$$\underset{\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}[{\hat{R}}_S(\mathbf{w})]-\underset{\mathbf{u}\in {\mathbb{R}}^d}{\min }{\hat{R}}_S(\mathbf{u})+\underset{\mathbf{u}\in {\mathbb{R}}^d}{\min }{\hat{R}}_S(\mathbf{u})-\underset{S}{E}[\hat{R}({\mathbf{w}}_{\lambda }^{\star })].$$

$$\underset{\mathbf{w}\sim {\hat{p}}_{S,\gamma }}{E}\left[{\hat{R}}_S(\mathbf{w})\right]-\underset{\mathbf{u}\in {\mathbb{R}}^d}{\min }{\hat{R}}_S(\mathbf{u})=\frac{1}{\gamma }\mathrm{tr}\left({\nabla }^2{\hat{R}}_S({\hat{\mathbf{w}}}_{\lambda }){\left({\nabla }^2{\hat{R}}_S({\hat{\mathbf{w}}}_{\lambda })+\lambda \mathbf{I}\right)}^{-1}\right)$$

$$-\gamma \underset{S}{E}\left[\underset{{\hat{p}}_{S,\gamma }}{E}\left[{\hat{R}}_{S,\lambda }(\mathbf{w})\mid \mathbf{w}\in {\mathcal{E}}^{\star }\left(r\right)\right]\right]+\frac{\gamma }{2}\underset{S}{E}\left[\underset{{\hat{p}}_{S,\gamma }}{E}\left[\Vert \mathbf{w}-{\mathbf{w}}_{\lambda }^{\star }{\Vert }_{{\mathbf{H}}_{\lambda }^{\star }}^2\mid \mathbf{w}\in {\mathcal{E}}^{\star }\left(r\right)\right]\right]$$

$$\displaystyle\lesssim\$$

Full text

1 Introduction

In the parametric setting of statistical learning, the learner is given a tuple $S=\left(z_{1},\ldots,z_{m}\right)$ of training examples, that are drawn independently of each other from a fixed and unknown probability distribution $\mathcal{D}$ supported on an example space $\mathcal{Z}$. Based on the training examples $S$ the learner selects a model $\boldsymbol{w}$ from a parameter space $\mathbb{R}^{d}$. The learner’s goal is to minimize the statistical risk $R(\boldsymbol{w})=\operatorname*{\mathbb{E}}_{z}[\ell(\boldsymbol{w},z)]$ of the selected model, where $z$ is drawn from $\mathcal{D}$ and $\ell\,:\,\mathbb{R}^{d}\times\mathcal{Z}\to[0,M]$ is some known loss function, which we assume to be non-negative, bounded, and twice differentiable.

A learner following the Empirical Risk Minimization (ERM) principle selects a model with the smallest empirical risk. Learners often also incorporate a penalty, leading to selecting a model from the set

[TABLE]

where $\widehat{R}_{S}(\boldsymbol{w})$ is the empirical risk of $\boldsymbol{w}$, defined by

[TABLE]

In the following, we will abbreviate the regularized empirical risk by $\widehat{R}_{S,\lambda}$ and its population counterpart, $\operatorname*{\mathbb{E}}[\widehat{R}_{S,\lambda}]$, by $R_{\lambda}$. In this paper we study a randomized version of ERM known as Gibbs-ERM. A Gibbs algorithm outputs a model $\boldsymbol{w}\in\mathbb{R}^{d}$ sampled from the Gibbs density

[TABLE]

where $Z=\int_{\mathbb{R}^{d}}e^{-\gamma\left(\widehat{R}_{S}(\boldsymbol{w})+\lambda\|\boldsymbol{w}\|^{2}\right)}\mathop{}\!\mathrm{d}\boldsymbol{w}$ is the normalization constant and $\widehat{R}_{S}$ is assumed to be such that $Z<\infty$ (for instance, this is the case when $\|\cdot\|$ is any norm and $\widehat{R}_{S}$ is nonnegative). It is not hard to see that we obtain ERM as a special case of (2) for $\gamma\to\infty$. In the following expectations $\operatorname*{\mathbb{E}}[\cdot]$ are taken with respect to the joint distribution over the sample space $\mathcal{Z}^{m}\times\mathbb{R}^{d}$ (i.e., the product of the example space and the parameter space) unless explicitly stated otherwise, for instance $\operatorname*{\mathbb{E}}_{\boldsymbol{w}\sim\widehat{p}_{S,\gamma}}[\cdot]$.

Gibbs-ERM reveals its usefulness when the regularized empirical risk is non-convex and (1) becomes intractable. This scenario brings out the connections between Gibbs-ERM and stochastic optimization algorithms, for instance Stochastic Gradient Langevin Dynamics (SGLD) —see below, along with a number of other settings in which Gibbs-ERM arises naturally. One tantalizing related line of research lies in understanding theoretical properties of learning in overparameterized problems, such as deep neural networks, through the prism of stochastic optimization, since in these settings Stochastic Gradient Descent (SGD) and its variants become de facto method of choice. We believe that Gibbs-ERM principle provides an opportunity for explaining some of the learning-theoretic phenomena in this area.

In this paper we focus on the statistical properties of the Gibbs-ERM by analyzing distribution-dependent excess risk bounds. In particular, we give upper bounds on the excess risk that can be much smaller compared to the previous literature, for instance SGLD (Raginsky et al., 2017), depending on the interplay between the data-generating distribution and the loss function.

SGLD algorithm.

The recent interest in stochastic gradient descent algorithms for non-convex optimization led to the study of a variant called SGLD. Apart from its simplicity, SGLD has amenable theoretical properties, such as asymptotic convergence to global minima and polynomial saddle-point escape times (Ge et al., 2015). The update rule of plain SGLD is

[TABLE]

where $\widehat{\boldsymbol{w}}_{1}$ is sampled from a fixed distribution, $\boldsymbol{\xi}_{t}$ is a standard Gaussian “noise” vector (independent from the choice of $\widehat{\boldsymbol{w}}_{1}$), and $\eta$ is a step size. The SGLD algorithm is known to approximate the continuous-time Langevin diffusion equation

[TABLE]

where $\boldsymbol{b}(t)$ is the standard Brownian motion. Indeed, under appropriate assumptions on the empirical risk, one can show that the solution to (3) admits (2) as a stationary distribution (Raginsky et al., 2017; Tzen et al., 2018). While convergence in the limit is reassuring, the best known bound on the mixing time is of order $\text{polylog}(1/\varepsilon)\,e^{\mathcal{\tilde{O}}(d)}$ for non-convex empirical risks assuming that objective is smooth and dissipative (roughly speaking, assuring that the process (3) on average moves towards the origin) (Raginsky et al., 2017; Xu et al., 2018), and it is not clear whether the exponential dependence on the parameters can be eliminated.

Information Risk Minimization.

Gibbs-ERM naturally arises when introducing a relative entropy regularization in the so-called Information Risk Minimization framework (Zhang, 2006; Xu and Raginsky, 2017). Indeed $\widehat{p}_{S,\gamma}$ in (2) can be equivalently defined as the solution to the following convex optimization problem

[TABLE]

where $\mathcal{M}_{1}$ is the set of all sample-dependent probability densities on $\mathbb{R}^{d}$, with sample drawn from $\mathcal{D}$, and $\mathrm{KL}$-divergence is defined between densities that are absolutely continuous with respect to some measure over $\mathbb{R}^{d}$. Problem (4) can be also motivated from perspective of the PAC-Bayesian analysis (McAllester, 1998; Seeger, 2002), where (2) is the density minimizing the bound on the expected risk. Another instance of (4) is the well-known Maximum Entropy Discrimination framework of Jaakkola et al. (1999).

Sampling from (2).

Markov chain Monte Carlo (MCMC) algorithms can be used to sample directly from (2). Unfortunately, this is often known to be computationally inefficient (Andrieu et al., 2003). On the other hand, there is a number of cases where MCMC demonstrates amenable computational properties, for instance when sampling from log-concave densities (Andrieu et al., 2003). Recent works have also showed that for a particular class of densities (such as smooth and strongly concave densities) variations of MCMC algorithms can sometimes achieve linear convergence (Cheng et al., 2018b), for certain non-log-concave densities MCMC variants can achieve polynomial convergence in the dimension (Cheng et al., 2018a), or ever achieve faster convergence than optimization algorithms (Ma et al., 2018). Another popular line of research is a variational approximations of the Gibbs density, such as Variational Bayes (Wang and Blei, 2018) where one resorts to the variational approximation of a target density.

1.1 Our Contribution

The algorithms discussed above perform randomized empirical risk minimization. However, minimizing empirical risk does not always lead to minimization of the risk. Hence, the quality of the solution $A(S)$ generated by the randomized algorithm $A$ given the training set $S$ is typically analyzed through the notion of excess risk $\operatorname*{\mathbb{E}}_{S,A}\left[R(A(S))\right]-R(\boldsymbol{w}^{\star})$, where $\boldsymbol{w}^{\star}$ is one of the minimizers of the risk. This is decomposed into the generalization error $R(A(S))-\widehat{R}_{S}(A(S))$ and the term $\widehat{R}_{S}(A(S))-R(\boldsymbol{w}^{\star})$. Similarly to Raginsky et al. (2017), we follow instead a Gibbs-centric decomposition of the excess risk

[TABLE]

The first term is due solely to the dynamics of the algorithm, be it SGLD or a sampling procedure, while the second one is a purely learning-theoretic quantity. Raginsky et al. (2017) mainly focused on the finite-time analysis of the first term for SGLD while showing convergence for non-convex objective functions. Their analysis of the second term —the statistical excess risk— provides a bound of order (ignoring logarithmic factors)

[TABLE]

where $\lambda_{\star}$ is a positive spectral gap characterizing the exponential convergence rate of the Langevin diffusion to the stationary point. They conservatively bounded the reciprocal of $\lambda_{\star}$ as

[TABLE]

This results in a statistical excess risk bound with a rather pessimistic exponential dependence on the ambient dimension $d$. Therefore, a natural question to ask is whether the dependence on $d$ can be improved by taking into account specific properties of the learning problem, and whether the dependence on $\lambda_{\star}$ can be avoided altogether (since the statistical excess risk does not really depend on the convergence properties of SGLD). We believe that $\lambda_{\star}$ (which has exponential dependence on the dimension) can be avoided in the analysis of Raginsky et al. (2017), although in their case this would not have improved the final result due to the contribution of the computational excess risk.

In this paper we consider the statistical excess risk, while we forego computational aspects of concrete algorithms. In particular, we focus on the distribution-dependent analysis of statistical excess risk (or, simply, excess risk). In the following we show upper bounds on the statistical excess risk that can be much smaller than (5) depending on the interplay between the data-generating distribution and the loss function.

The notion of excess risk considered in this paper is defined with respect to the regularized minimizer of the risk

[TABLE]

Note that this is not a limitation because we can always recover the regularizer-free analysis by looking at the asymptotic behavior $\lambda\rightarrow 0$. In the following, we assume that both risk and empirical risk are bounded and twice differentiable, and the Hessian matrix of the risk is locally-Lipschitz (in a sense precisely defined later on). Therefore, the objective function of (7) (as well as the one of (1)) can have more than one minimum. However we assume that local minima are isolated, meaning that a sufficiently small neighborhood of $\boldsymbol{w}^{\star}_{\lambda}$ contains a unique minimum. One compelling example is a large family of non-convex strict-saddle ERM problems (Ge et al., 2015; Gonen and Shalev-Shwartz, 2017), such as matrix completion, tensor decomposition, PCA, ICA, and others. Another example of such is an empirical risk of a ReLU neural network with weight decay, or L2 regularization, (Milne, 2018, Theorem 1) where minima resulting in a sufficiently small empirical risk are locally strongly-convex. Even though the theorem holds for empirical measures, we suspect that it could be extended to the population risk through the uniform convergence argument.

Localized excess risk.

Before delving into the global analysis of the excess risk, we look at the local approximation properties of the Gibbs-ERM principle, which will also be instrumental in the forthcoming global analysis. We begin by looking at the localized excess risk with respect to a fixed minimizer $\boldsymbol{w}^{\star}_{\lambda}$. Specifically, we consider the excess risk of a parameter $\boldsymbol{w}$ generated by Gibbs-ERM within a certain neighborhood around $\boldsymbol{w}^{\star}_{\lambda}$. This is defined as

[TABLE]

where conditioning is taken on the event that $\boldsymbol{w}$ lies in the ellipsoid $\mathcal{E}^{\star}\left(r\right)$ of radius $r$ centered at the minimizer $\boldsymbol{w}^{\star}_{\lambda}$ and aligned with the curvature of the risk at that minimum. We prove (Theorem 4.3) that the local excess risk behaves as111Throughout this paper, we use $f\lesssim g$ to say that there exists a universal constant $C>0$ such that $f\leq Cg$ holds uniformly over all arguments.

[TABLE]

where $\boldsymbol{H}^{\star}$ is the Hessian matrix $\nabla^{2}R(\boldsymbol{w}^{\star}_{\lambda})$, $\boldsymbol{H}^{\star}_{\lambda}=\boldsymbol{H}^{\star}+2\lambda\boldsymbol{I}$, and $\varepsilon(r)$ is a local approximation error that vanishes as $r\to 0$ (defined precisely in Section 4). The trace term in (8), a distribution-dependent quantity known as the effective dimension of minimizer $\boldsymbol{w}^{\star}_{\lambda}$, can be also expressed as $\lambda_{1}/(\lambda_{1}+\lambda)+\cdots+\lambda_{d}/(\lambda_{d}+\lambda)$ where $\lambda_{1},\ldots,\lambda_{d}$ are the eigenvalues of $\boldsymbol{H}^{\star}$. This can be viewed as a “soft” version of the rank of $\boldsymbol{H}^{\star}$. Note that $\mathrm{tr}\left(\boldsymbol{H}^{\star}\boldsymbol{H}^{\star-1}_{\lambda}\right)\leq d$ always, and $\mathrm{tr}\left(\boldsymbol{H}^{\star}\boldsymbol{H}^{\star-1}_{\lambda}\right)\ll d$ whenever the spectrum of the Hessian matrix is light-tailed. This notion of effective dimension occasionally appears in the analysis of ridge regression (Audibert and Catoni, 2011; Neu and Rosasco, 2018).

Next, to get a sense of the strength of the bound and as a sanity check, one may look at limiting cases with respect to parameters $\lambda$ and $\gamma$. When $\lambda\to 0$, corresponding to the unregularized Gibbs-ERM principle, our bound becomes

[TABLE]

where $\lambda_{\mathrm{min}}(\boldsymbol{H}^{\star})$ denotes the smallest non-zero eigenvalue of $\boldsymbol{H}^{\star}$. Assuming the radius is set to $r=\gamma^{-\frac{1+p}{3}}$ for some $p>0$, the polynomial term in the right-hand side of the above bound vanishes as $\gamma,m\to\infty$, even for singular $\boldsymbol{H}^{\star}$. On the other hand, for $\lambda>0$ and $\gamma,m\to\infty$ the right-hand side of (8) tends to zero, and the bound backs up the intuition that the Gibbs-ERM principle should exactly recover the ERM solution. This observation also serves as a sanity check that the bound is reasonably tight, at least with respect to $\gamma$.

Finally, in cases when the Hessian matrix of the risk is constant, for instance in Regularized Least Squares (RLS) problems, $\varepsilon(r)=0$ and our bound specializes to

[TABLE]

When $\gamma$ is tuned optimally the above bound becomes $\Delta(\boldsymbol{w}^{\star}_{\lambda})\lesssim m^{-\frac{1}{4}}\sqrt{\mathrm{tr}\left(\boldsymbol{H}^{\star}\boldsymbol{H}^{\star-1}_{\lambda}\right)}$. Note that, for the square loss, the best known dependence on $m$ that can be achieved is $m^{-\frac{1}{2}}$. The worse exponent in our bound is the price we pay for the generality of our approach. Although our results are never worse in terms of the dimension, the dependence on the sample size in our bounds is worse than in those of (Raginsky et al., 2017, (3.27)). This is because we are stating bounds in terms of the distribution-dependent effective dimension. We can obtain the same rate as Raginsky et al. (2017) by expressing the effective dimension in terms of the empirical risk.

Global excess risk.

Next, we consider a global notion of excess risk,

[TABLE]

Here, in the second term $\pi$ is a distribution over the countable set $\mathcal{I}$ of all minima (recall that minima are isolated). In this setting, all minima, Hessian matrices, approximation errors, and ellipsoids gain a corresponding subscript $i\in\mathcal{I}$: $\boldsymbol{w}^{\star}_{\lambda,i}$, $\boldsymbol{H}^{\star}_{\lambda,i}$, $\varepsilon_{i}(r)$, and $\mathcal{E}^{\star}_{i}(r)$.

We first focus on the finite-temperature distribution over minima of the regularized risk

[TABLE]

where probabilities are taken with respect to the population Gibbs density $p_{\gamma}(\boldsymbol{w})\propto e^{-\gamma R_{\lambda}(\boldsymbol{w})}$. For this distribution we prove that

[TABLE]

for any $r<r_{0}$ where the radius $r_{0}$ is chosen such that all ellipsoids in the set \big{\{}\mathcal{E}^{\star}_{i}(r)\,:\,i\in\mathcal{I}\big{\}} are disjoint, $\mathcal{C}^{\star}(r)$ is the complement of the union of the ellipsoids in this set (i.e., the volume outside of minima), and the expectation is taken with respect to $I\sim\pi_{\gamma,r}$.

Note that there is a trade-off in (10) between the first term, which is essentially a bound on the expected excess risk in the neighborhood of a minimum drawn according to $\pi_{\gamma,r}$, and the last term, which is the probability of sampling outside of the neighborhood of any minimum. This means that we can obtain an oracle inequality by choosing $r\in[0,r_{0}]$ such that it minimizes the right-hand side of (10).

Now we focus on the probability of the complement, which behaves as

[TABLE]

where $\alpha_{d/2}$ depends only on $d$. So, as long as $r^{2}\gamma$ is increasing and $\gamma\varepsilon_{i}(r)$ is non-increasing, the probability of generating a solution outside of the minima decreases. For example, when $r=\gamma^{\frac{p-1}{2}}$ for $p\in(0,1/3]$ (as discussed in Section 4.2) the right-hand side of (11) vanishes as $\gamma\to\infty$.

Asymptotic pseudo excess risk.

It is also natural to ask what happens in the zero-temperature regime $\gamma\to\infty$, when the Gibbs-ERM principle reduces to a rule for selecting empirical risk minimizers. We can study this by observing that (10) vanishes when the radius is set to $r=\gamma^{\frac{p-1}{2}}$ —as we previously discussed— and $\gamma$ is set to $m^{\frac{1}{4}}$, which is a meaningful result. Indeed, whenever $m=\infty$, then $\gamma=\infty$ and the risk of Gibbs-ERM should not differ from the risk of a minimum drawn from the limiting distribution $\pi_{\infty}=\lim_{\gamma\rightarrow\infty}\pi_{\gamma,r}$. Interestingly, the distribution $\pi_{\infty}$ has the following analytic form (this is shown in Lemma 4.4 assuming the tuning $r=\gamma^{\frac{p-1}{2}}$):

[TABLE]

where $\mathcal{I}^{\text{{glob}}}$ is a countable set enumerating global minima of the regularized risk. Hence, the probability of a minimum $\boldsymbol{w}^{\star}_{\lambda,i}$ is proportional to the reciprocal of the normalized volume of the ellipsoid defined by the eigenvalues of the Hessian at that minimum. In particular, this implies that the probability of choosing a global minimum with larger volume is higher. Note that all suboptimal minima have zero probability under $\pi_{\infty}$. At the same time, in this asymptotic regime it is also rather clear that Gibbs-ERM generates models outside of the neighborhoods of the minima with zero probability. These two observations show how to strike a middle ground between the nonasymptotic bound of (10) and the asymptotic distribution (12). This is captured by the global asymptotic pseudo excess risk

[TABLE]

which bounds the localized excess risk at finite temperature $\gamma>0$ when minima are drawn from the global limiting distribution $\pi_{\infty}$. For any $r\geq 0$ we have

[TABLE]

Observe that whereas the local excess risk (8) is essentially controlled by the soft rank of the minimum, the bound (13) implies that globally this is not necessarily the case, since low-rank minima have smaller probability under distribution $\pi_{\infty}$.

1.2 Additional Related Work

Generalization bounds for the Gibbs-ERM principle has been extensively studied in a number of works over the past years. One prolific thread of research in this direction is the PAC-Bayesian analysis starting from seminal works of McAllester (1998); Langford and Shawe-Taylor (2003) until the latest developments, for instance (Germain et al., 2015; Grünwald and Mehta, 2017). PAC-Bayesian analysis follows the uniform convergence argument (over the class of densities), where, as was pointed out earlier Gibbs density is the one minimizing the bound on the risk (Information Risk Minimization). In this paper we focus on the excess risk bounds rather than generalization bounds, that manifest distribution-dependent properties of a potentially non-convex risk. PAC-Bayesian excess risk bounds have also been studied in a number of contexts (Alquier et al., 2016; Audibert and Catoni, 2011; Grünwald and Mehta, 2017), however, these typically assume convex risk (e.g. least-squares), or focus on the properties of the hypothesis class (sometimes distribution-dependent) rather than those of the objective function (Grünwald and Mehta, 2017). Distribution-dependent arguments have been exploited to develop sharper generalization bounds (Lever et al., 2013), and data-dependent PAC-Bayesian bounds were shown to be numerically non-vacuous as was shown by Dziugaite and Roy (2018).

A number of works have also analyzed generalization and approximation properties of Gibbs-ERM from the algorithmic point of view. A heuristic approach to analysis of SGLD algorithm was given by Welling and Teh (2011); Mandt et al. (2016) while recent works have also argued that its generalization ability is controlled by the “width” (or the notion of pseudo-rank) at the minimum of the empirical risk (Keskar et al., 2017; Chaudhari and Soatto, 2018; Liang et al., 2019), which is reminiscent of the effective dimension studied in this paper. Mou et al. (2018) developed generalization bounds for SGLD from PAC-Bayesian and algorithmic stability point of view (Bousquet and Elisseeff, 2002). Apart from the excess risk bounds, Raginsky et al. (2017) also showed Gibbs-ERM specific generalization bounds through the algorithmic stability framework. In this paper we also analyze generalization and stability of Gibbs-ERM principle, however we present a simpler proof technique similar in spirit to Xu and Raginsky (2017). Finally, Sheth and Khardon (2017) analyzed a slightly different notion of excess risk bounds for the variational inference assuming the use of latent Gaussian models (such as generalized linear models and Gaussian processes).

2 Preliminaries

Throughout this paper, we use $f\lesssim g$ to indicate that there exists a universal constant $C>0$ such that $f\leq Cg$ holds uniformly over all arguments. Let $\mathcal{B}_{r}(\boldsymbol{z})\subset\mathbb{R}^{d}$ be the ball of center $\boldsymbol{z}$ and radius $r>0$ and let $\mathcal{B}_{r}=\mathcal{B}_{r}(\boldsymbol{0})$. Given a positive definite $d\times d$ matrix $\boldsymbol{M}$, define $\|\boldsymbol{x}\|_{\boldsymbol{M}}^{2}=\boldsymbol{x}^{\top}\boldsymbol{M}\boldsymbol{x}$ for $\boldsymbol{x}\in\mathbb{R}^{d}$. Then, for any positive semi-definite $d\times d$ matrix $\boldsymbol{A}$ and $r>0$ the corresponding ellipsoid centered at $\boldsymbol{x}_{0}\in\mathbb{R}^{d}$ is defined as $\mathcal{E}(\boldsymbol{x}_{0},\boldsymbol{A},r)\equiv\left\{\boldsymbol{x}\in\mathbb{R}^{d}~{}:~{}\|\boldsymbol{x}_{0}-\boldsymbol{x}\|_{\boldsymbol{A}}\leq r\right\}$.

If $p$ and $q$ are densities that are absolutely continuous with respect to a measure $\mu$ over $\mathbb{R}^{d}$, the Kullback-Liebler (KL) divergence between $p$ and $q$ is defined as

[TABLE]

3 Sketch of the Analysis

In this section we briefly explain the main arguments at the basis of our analysis. We start from the analysis of the localized excess risk, which is decomposed into the generalization error (i.e., the difference between the risk and empirical risk) and the gap between the empirical risk and the risk of the minimizer $\boldsymbol{w}^{\star}_{\lambda}$,

[TABLE]

where the expectation is taken with respect to the empirical Gibbs density (2). For brevity we omit the indices corresponding to the minima, as in the localized setting we consider a local minimum at the time. The generalization error of Gibbs-ERM is captured by Theorem 3.1 below (whose proof can be found in Section 5.2).

Theorem 3.1** (Generalization bound).**

Consider any loss function $f:\mathbb{R}^{d}\times\mathcal{Z}\to\mathbb{R}$ that is $\sigma$-sub-Gaussian in the first argument with respect to the Gibbs density

[TABLE]

conditioned on a measurable $A\subseteq\mathbb{R}^{d}$. Then the generalization error of Gibbs-ERM satisfies

[TABLE]

Assuming that $f$ is everywhere bounded by $M$, Hoeffding Lemma implies the bound $\frac{M^{2}\gamma}{2m}$. Under the same boundedness assumption, a bound with similar rates was also shown by Xu and Raginsky (2017) within the mutual information framework. Our proof works by showing that the Gibbs density is on-average replace-one stable in the sense of (Shalev-Shwartz and Ben-David, 2014, Section 13) (we include it here for completeness).

The second quantity (15) is less straightforward to control in a distribution-dependent setting. Raginsky et al. (2017) give a global upper bound

[TABLE]

by further decomposing (15) as follows

[TABLE]

The second term is bounded trivially, while the analysis of the first term follows the so called “almost ERM” argument. In other words, understanding how “close” the solutions generated by Gibbs-ERM are to the solutions of ERM.

In our distribution-dependent setting we follow a different route: consider the case of RLS, where the empirical risk is an average of square-regularized losses. A rather straightforward argument (based on Gaussian integration) gives the following identity in an “almost ERM” style:

[TABLE]

where $\widehat{\boldsymbol{w}}_{\lambda}$ is a minimizer of the RLS problem. Observe that in the above identity we obtain an empirical counterpart of the effective dimension introduced in (8). Since our goal is a distribution-dependent result, one possibility is to consider the concentration of Hessian eigenvalues. However, we follow a more direct approach. As we said earlier, the “almost ERM” analysis of RLS is relatively easy using Gaussian integration (since $\widehat{p}_{S,\gamma}$ is a Gaussian density). Since we deal instead with general smooth densities, our idea is to quantify the gap between the density at hand and the Gaussian density. This is nicely handled by the transportation lemma (Boucheron et al., 2013, Lemma 4.18), characterizing the difference between the expectations of different densities in terms of the KL-divergence between them. As a comparison we choose the Gaussian density q_{\gamma}\propto\exp\big{(}-\frac{\gamma}{2}\|\boldsymbol{w}-\boldsymbol{w}^{\star}_{\lambda}\|_{\boldsymbol{H}^{\star}_{\lambda}}^{2}\big{)}, which only depends only on distribution-related quantities. An “almost ERM” style analysis applied to $q_{\gamma}$ will give us exactly a distribution-dependent effective dimension. So, the only issue is the actual gap between densities. By expanding the KL-divergence we observe that the critical terms are

[TABLE]

To obtain (17), instead of using concentration, we resort to the generalization bound of Theorem 3.1, whereas (18) is obtained by Taylor expansion of the regularized risk around its minimizer. This is formally shown in Lemma 4.2, while the decomposition (14) is bounded in Theorem 4.3. Hence, the gap is quantified by the approximation error at the radius $r$ plus a sample-dependent term due to the use of empirical Gibbs density. These terms appear in excess risk bounds (8), (10), and (13).

3.1 Global Analysis

Starting from the conditional local excess risk in the form given by the left-hand side of (14), we analyze a notion of global risk by bounding the probability of individual ellipsoids (i.e., neighborhoods of minima) and that of the complement of their union. Since for $\gamma\to\infty$ the probability of a complement approaches zero (as discussed in Remark 4.6), in order to obtain an asymptotic bound it is enough to focus on the relative probability of ellipsoids. In Lemma 4.4 we derive upper and lower bounds on this probability via Laplace approximation (Lemma 5.11), and then analyze their limit for $\gamma\to\infty$. Combining the local excess risk bound and the bound on the asymptotic relative probability of ellipsoids allows us to control the asymptotic global pseudo excess risk (13) —see Corollary 1.

Finally, using a nonasymptotic bound on the probability of a complement —see the proof of Theorem 4.5— we can apply the law of total expectation to get also a nonasymptotic bound on the global excess risk.

4 Main results

4.1 Local analysis

We first turn our attention to the local analysis considering a fixed minimizer222We will drop subscript indexing of minima in this section.

[TABLE]

Specifically, we prove that the risk of Gibbs-ERM in a neighborhood of $\boldsymbol{w}^{\star}_{\lambda}$ is controlled by the local effective dimension $\mathrm{tr}\left(\boldsymbol{H}^{\star}\boldsymbol{H}^{\star-1}_{\lambda}\right)$, defined in terms of the Hessian $\boldsymbol{H}^{\star}=\nabla^{2}R(\boldsymbol{w}^{\star}_{\lambda})$ of the risk of the minimizer, where $\boldsymbol{H}^{\star}_{\lambda}=\boldsymbol{H}^{\star}+2\lambda\boldsymbol{I}$. We require that Hessians do not change “too quickly” by assuming that Hessian of the risk is Lipschitz in an ellipsoid $\mathcal{E}^{\star}(r)=\mathcal{E}(\boldsymbol{w}^{\star}_{\lambda},\boldsymbol{H}^{\star}_{\lambda},r)$ centered at the minimizer and aligned with the local curvature. Formally the local Lipschitzness of the Hessian is defined as follows.

Definition 4.1** (Locally-Lipschitz Hessian).**

The Hessian $\nabla^{2}R$ is locally Lipschitz around a minimizer $\boldsymbol{w}^{\star}_{\lambda}$ if there exists a function $L^{\star}:\mathbb{R}_{+}\to\mathbb{R}_{+}$ such that

[TABLE]

Note that local Lipschitzness of the risk Hessian implies local Lipschitzness of the regularized risk Hessian for the same function $L^{\star}$.

Local Lipschitzness of Hessians plays an important role in bounding the gap between the Gibbs density and the Gaussian density, as discussed in Section 3. In particular, the approximation error introduced by taking a Taylor expansion of the regularized risk up to the third term is

[TABLE]

where $\lambda_{\mathrm{min}}$ is the smallest non-zero eigenvalue of $\boldsymbol{H}^{\star}$. Observe that $\lim_{r\rightarrow\infty}\varepsilon(r)=0$ because $L^{\star}(r)$ converges to $\|\nabla^{2}R(\boldsymbol{w}^{\star}_{\lambda})\|_{2}$, and that $\varepsilon(r)=0$ for any constant Hessian matrices (e.g., in the case of RLS). Lemma 4.2 below here establishes a result needed to prove our bound on the local excess risk.

Lemma 4.2**.**

For any minimizer $\boldsymbol{w}^{\star}_{\lambda}$ of the regularized risk we have

[TABLE]

(The proof of Lemma 4.2, along with that of all remaining statements in this section, can be found in Section 5.3.) This lemma, combined with the bound on the generalization error (Theorem 3.1), gives the desired result.

Theorem 4.3** (Localized Excess Risk Bound).**

Assume the same as in Lemma 4.2. Then,

[TABLE]

4.2 Global analysis

We now turn our attention to the global analysis of the excess risk. Since we deal with a countable set of local minima (indexed by $\mathcal{I}$), we add a subscript to all minima-dependent quantities, such as $\boldsymbol{w}^{\star}_{\lambda,i}$, $\boldsymbol{H}^{\star}_{\lambda,i}$, $\mathcal{E}^{\star}_{i}(r)$. In particular, the approximation error is now defined as

[TABLE]

where $L^{\star}_{i}$ is the local Lipschitz constant with respect to the minimum $\boldsymbol{w}^{\star}_{\lambda,i}$, and $\lambda_{\mathrm{min},i}$ is the smallest non-zero eigenvalue of the Hessian matrix $\boldsymbol{H}^{\star}_{\lambda,i}$.

Next, we introduce an important assumption on the geometry of the regularized risk around its minimizers.

Assumption 1

All local minima $\boldsymbol{w}^{\star}_{\lambda}\in\operatorname*{arg\,min}_{\boldsymbol{w}\in\mathbb{R}^{d}}R_{\lambda}(\boldsymbol{w})$ satisfy $\nabla R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda})=\boldsymbol{0}$ and are such that $\nabla^{2}R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda})$ is positive definite. In other words all local minima are isolated.

The above assumption implies that there exists a number $r_{0}>0$ such that

[TABLE]

In other words, ellipsoids centered at minimizers and aligned with the local curvature of $R_{\lambda}$ are non-overlapping. In addition to the set $\mathcal{I}$, indexing minima of the regularized risk, let $\mathcal{I}^{\text{{glob}}}\subseteq\mathcal{I}$ index the global minima and denote its complement by $\mathcal{I}^{\text{{subopt}}}\equiv\mathcal{I}\setminus\mathcal{I}^{\text{{glob}}}$. Finally, introduce the complement of the ellipsoids centered at the minima (later called, with some abuse of terminology, complement of the minima),

[TABLE]

The first result in this section concerns the distribution of local minima. In particular, we give an upper bound on the relative probability $\pi_{\gamma,r}(i)$ of the $i$-th minimum, and then derive the analytic form of the asymptotic distribution $\pi_{\infty}$.

Lemma 4.4** (Distribution of Minima).**

For all $r>0$,

[TABLE]

Moreover, assuming without loss of generality that $R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda,i})=0$ for all $i\in\mathcal{I}^{\text{{glob}}}$, and setting $r=\gamma^{-p}$ for $p>0$, we have

[TABLE]

We are now ready to state the main result of this section.

Theorem 4.5** (Global Excess Risk Bound).**

Assume the same as in Lemma 4.2. Then for any $r\in[0,r_{0}]$ the global excess risk satisfies

[TABLE]

where the expectation is taken with respect to $I\sim\pi_{\gamma,r}$ and the probability of the complement of the minima is bounded as

[TABLE]

with

[TABLE]

Remark 4.6**.**

We compute the value of $r$ approximately minimizing the the right-hand side in Theorem 4.5. Note that the probability of the complement of the minima decreases if we ensure that $r^{2}\gamma$ increases in $\gamma$ and $\gamma\varepsilon_{i}(r)\propto r^{3}\gamma$ is non-increasing. For instance we may set $r^{2}\gamma=\gamma^{p}$ for $p>0$ so that $r^{3}\gamma=\gamma^{1+\frac{3}{2}(p-1)}$. Hence we require $1+\frac{3}{2}(p-1)\leq 0$ which is satisfied for any $p\in(0,1/3]$. This implies that when $r=\gamma^{\frac{p-1}{2}}$ and $p\in(0,1/3]$ the probability of the complement of the minima and the approximation terms $\gamma\varepsilon_{i}(r),\varepsilon_{i}(r)$ all vanish as $\gamma\to\infty$.

Finally, combining the localized excess risk bound in Theorem 4.3 with Lemma 4.4 allows us to prove the following result about the asymptotic pseudo excess risk.

Corollary 1**.**

Assume the same as in Lemma 4.2. Then, for any $r>0$, the global asymptotic pseudo-excess risk satisfies

[TABLE]

where $I$ is distributed according to

[TABLE]

5 Proofs

5.1 Common Tools

We compute the Taylor expansion of $R_{\lambda}(\boldsymbol{w})$ for $\boldsymbol{w}\in\mathcal{E}(\boldsymbol{w}^{\star}_{\lambda},\nabla^{2}R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda}),r)$, where $\boldsymbol{w}^{\star}_{\lambda}$ is a minimizer of the regularized risk,

[TABLE]

where $\varepsilon(r)$ is defined in (21), and (25) follows because $\sqrt{\lambda_{\mathrm{min}}+\lambda}\|\boldsymbol{w}-\boldsymbol{w}^{\star}_{\lambda}\|\leq r$ where $\lambda_{\mathrm{min}}$ is the smallest non-zero eigenvalue of $\nabla^{2}R(\boldsymbol{w}^{\star}_{\lambda})$. In a similar way we have the upper expansion

[TABLE]

We now introduce a crucial transportation lemma which is instrumental in the following proofs.

Lemma 5.1** ((Boucheron et al., 2013, Lemma 4.18)).**

Let $Z$ be a real-valued integrable random variable with distribution $P$ such that

[TABLE]

for some $\sigma>0$ and let $Z^{\prime}$ be another random variable with distribution $Q$. If $Q$ is absolutely continuous with respect to $P$ and such that $\mathrm{KL}(Q\ ||\ P)<\infty$, then $\operatorname*{\mathbb{E}}[Z^{\prime}]-\operatorname*{\mathbb{E}}[Z]\leq\sqrt{2\sigma^{2}\mathrm{KL}\left(Q\ ||\ P\right)}$.

Next, we prove a helpful lemma about the log-ratio of Gibbs integrals.

Lemma 5.2**.**

Let $f_{A},f_{B}:\mathcal{X}\to\mathbb{R}$ such that

[TABLE]

is finite for all $\gamma>0,\mathcal{B}\subseteq\mathcal{X}$ and let

[TABLE]

where $f_{B}$ is similarly defined. Whenever $Z_{A}>0$ we have that

[TABLE]

Proof 5.3**.**

Observe that

[TABLE]

Since $-\ln()$ is a convex function, by Jensen’s inequality we obtain the desired result.

5.2 Generalization Bound for Gibbs-ERM

We start by proving a generalization bound for Gibbs-ERM.

Theorem 3.1 (restated)

Consider any loss function $f:\mathbb{R}^{d}\times\mathcal{Z}\to\mathbb{R}$ that is $\sigma$-sub-Gaussian in the first argument with respect to the Gibbs density

[TABLE]

conditioned on a measurable $A\subseteq\mathbb{R}^{d}$. Then the generalization error of Gibbs-ERM satisfies

[TABLE]

Proof 5.4**.**

Consider the training examples $S$ drawn i.i.d. from $\mathcal{D}$ and, for $i=1,\ldots,m$, denote by ${S^{(i)}}=\left\{z_{1},\ldots,z_{i-1},z,z_{i+1},\ldots,z_{m}\right\}$ a replace-one training data, where $z$ is independently drawn from $\mathcal{D}$. Throughout the proof, we drop $\gamma$ from the notation for the Gibbs density $\widehat{p}_{S,\gamma}$. Introduce the conditional Gibbs densities $\widehat{p}_{S\mid A}(\boldsymbol{w})$ and $\widehat{p}_{{S^{(i)}}\mid A}(\boldsymbol{w})$. We denote by $\operatorname*{\mathbb{E}}_{\widehat{p}_{S\mid A}}[\cdot]$ and $\operatorname*{\mathbb{E}}_{\widehat{p}_{{S^{(i)}}\mid A}}[\cdot]$ expectations with respect to $\widehat{p}_{S\mid A}$ and $\widehat{p}_{{S^{(i)}}\mid A}$. We start by rewriting the expected generalization error as

[TABLE]

Now we bound (30) showing the average replace-one stability of Gibbs distribution. We use the transportation Lemma 5.1 with $Q=\widehat{p}_{S\mid A}$ and $P=\widehat{p}_{{S^{(i)}}\mid A}$ we get that

[TABLE]

Next, we focus on $\mathrm{KL}$-divergence,

[TABLE]

By taking expectation with respect to $S$ and $z$ on both sides, we get that the first term in (36) can be expressed as

[TABLE]

where we could switch $z_{i}$ and $z$ on the right-hand side because their are both independently drawn from $\mathcal{D}$. Thus, the expectation of (31) with respect to $S$ and $z$ is upper-bounded as

[TABLE]

Solving the above with respect to the term on the left-hand side we get that for any $i=1,\ldots,m$,

[TABLE]

Substituting the above into (29) gives the desired generalization bound.

5.3 Localized Excess Risk Bounds

First, we prove a key lemma about the conditional expectation of quadratic forms.

Lemma 5.5**.**

Suppose that $\boldsymbol{x}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{M})$. Then for the ellipsoid

[TABLE]

and for any Positive Semi-Definite (PSD) $d\times d$ matrix $\boldsymbol{A}$ we have that

[TABLE]

where $F_{k}$ is the CDF of a $\mathcal{X}^{2}$-distribution with $k$ degrees of freedom.

Moreover the above implies that

[TABLE]

Proof 5.6**.**

Observe that

[TABLE]

where $\boldsymbol{\tilde{M}}$ is the covariance matrix of the Gaussian density $\mathcal{N}(\boldsymbol{0},\boldsymbol{M})$ conditioned on $\mathcal{E}(r)$. Next, we apply a result about moments of multivariate Gaussian densities under elliptical truncation (Tallis, 1963, p. 941) to get that

[TABLE]

where $F_{d}$ is a CDF of a $\mathcal{X}^{2}$ distribution. This proves the first identity.

The inequality is proven by expanding $\boldsymbol{\tilde{M}}$ further in terms of the Gamma function $\Gamma(\cdot)$ and the incomplete Gamma function $\gamma(\cdot,\cdot)$:

[TABLE]

Finally, we look at the limit of the ratio in the right-hand side of (44) as $r\rightarrow 0$. By L’Hôpital’s rule,

[TABLE]

concluding the proof.

Recall that $\mathcal{E}^{\star}\left(r\right)\equiv\mathcal{E}(\boldsymbol{w}^{\star}_{\lambda},\boldsymbol{H}^{\star}_{\lambda},r)$ is the ellipsoid of radius $r$ centered at $\boldsymbol{w}^{\star}_{\lambda}$.

Lemma 4.2 (restated)

For any minimizer $\boldsymbol{w}^{\star}_{\lambda}$ of the regularized risk we have

[TABLE]

Proof 5.7**.**

We abbreviate the regularized empirical risk by $\widehat{R}_{S,\lambda}(\boldsymbol{w})=\widehat{R}_{S}(\boldsymbol{w})+\lambda\|\boldsymbol{w}\|^{2}$ and recall that the regularized risk is denoted by $R_{\lambda}(\boldsymbol{w})=R(\boldsymbol{w})+\lambda\|\boldsymbol{w}\|^{2}$. Throughout the proof, we drop $\gamma$ from the notation for the Gibbs densities $\widehat{p}_{S,\gamma}$. Let $\widehat{p}_{S\mid\mathcal{E}^{\star}}$ be the Gibbs density (2) conditioned on the ellipsoid $\mathcal{E}^{\star}\left(r\right)$. Similarly, let $q_{\mid\mathcal{E}^{\star}}$ be the the Gaussian density

[TABLE]

conditioned on $\mathcal{E}^{\star}\left(r\right)$.

We begin by observing that $\widehat{R}_{S}$ is trivially $M^{2}/8$-sub-Gaussian since the loss function is bounded by $M$. Hence, by the transportation Lemma 5.1,

[TABLE]

where the last inequality is obtained by Jensen’s inequality. The KL term can be written as follows

[TABLE]

Now we relate the regularized empirical risk (47) to the regularized risk. By applying Theorem 3.1 with $A\equiv\mathcal{E}^{\star}(r)$ we get

[TABLE]

Using this result we can write

[TABLE]

where (48) is obtained by applying the lower Taylor expansion (26) to $\boldsymbol{w}\in\mathcal{E}^{\star}\left(r\right)$ . Now we bound the expected log-ratio term in (49) as

[TABLE]

where the last inequality is derived from the upper Taylor expansion (27). Substituting the above into (49) gives

[TABLE]

Now we go back to (45) and, using the upper Taylor expansion (27), we get

[TABLE]

where (53) vanishes since the first moment of elliptically-truncated Gaussian is zero (Tallis, 1963). Finally, we bound the first term in (54) by invoking Lemma 5.5. By taking $\boldsymbol{M}=\gamma\boldsymbol{H}^{\star}_{\lambda}$, $\boldsymbol{A}=\boldsymbol{H}^{\star}$, and $\boldsymbol{x}=\boldsymbol{w}-\boldsymbol{w}^{\star}_{\lambda}$, and using Lemma 5.5 we get

[TABLE]

and

[TABLE]

Now, combining these results with the bound on KL-divergence (52), and substituting into (46), gives the stated result.

Theorem 4.3 (restated)

Assume the same as in Lemma 4.2. Then,

[TABLE]

Proof 5.8**.**

From the definition of local generalization error,

[TABLE]

where the last inequality is derived from Theorem 3.1 and Lemma 4.2.

5.4 Statements about Probability Mass of Ellipsoids

Before we prove our bound on the global excess risk, we introduce some necessary technical notions about the regularized gamma function, which can be interpreted as the probability of an Euclidean ball of radius $z$ under a Gaussian density with covariance matrix $\boldsymbol{I}$.

Theorem 5.9** ((NIST, 2018, Regularized Gamma Function)).**

Denote the regularized gamma function by

[TABLE]

where $\Gamma(a,z)$ is the upper incomplete Gamma function given by

[TABLE]

Then, for all $z\geq 0$ and $a>0$,

[TABLE]

where

[TABLE]

with equality in (55) only when $a=1$.

Proposition 2** (Truncated Gaussian Integrals).**

For any $\gamma,r>0$,

[TABLE]

where $\mathcal{B}(r)$ is the $d$-dimensional Euclidean ball. In addition, for any $d\times d$ semi-definite matrix $\boldsymbol{A}$,

[TABLE]

Proof 5.10**.**

By the integration of radial functions

[TABLE]

In addition we have

[TABLE]

where the third step is obtained through the change of variables $\mathop{}\!\mathrm{d}\boldsymbol{A}^{\frac{1}{2}}\boldsymbol{u}=\mathop{}\!\mathrm{d}\boldsymbol{v}$.

Recall that $\mathcal{E}^{\star}(r)\equiv\mathcal{E}(\boldsymbol{w}^{\star}_{\lambda},\boldsymbol{H}^{\star}_{\lambda},r)$, where $\boldsymbol{H}^{\star}_{\lambda}=\nabla^{2}R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda})$.

Lemma 5.11** (Bounds on the Ellipsoid probability mass.).**

Let $\boldsymbol{w}^{\star}_{\lambda}$ be any minimizer of $R_{\lambda}$. Then the following results hold for probabilities of ellipsoids under the density $e^{-\gamma R_{\lambda}(\boldsymbol{w})}/Z$,

[TABLE]

Proof 5.12**.**

By applying the lower Taylor expansion (26) in the exponent of the Gibbs density we get

[TABLE]

where (57) is obtained via the change of variables $\boldsymbol{u}=\boldsymbol{H}^{\star\frac{1}{2}}_{\lambda}(\boldsymbol{w}-\boldsymbol{w}^{\star}_{\lambda})$ and (58) via Proposition 2. This shows the first result. The second result follows in a similar way exploiting the upper Taylor expansion (27),

[TABLE]

Finally, we give a lower bound on the probability of $\mathcal{E}^{\star}(r)$. We start by upper bounding the normalization constant using the lower Taylor expansion (26),

[TABLE]

Combining the above with (59) gives \mathbb{P}\big{(}\mathcal{E}^{\star}(r)\big{)}\geq e^{-\frac{\gamma}{3}\varepsilon(r)}P\left(\frac{d}{2},\frac{r^{2}\gamma}{2}\right) thus completing the proof.

Lemma 4.4 (restated)

For all $r>0$,

[TABLE]

Moreover, assuming without loss of generality that $R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda,i})=0$ for all $i\in\mathcal{I}^{\text{{glob}}}$, and setting $r=\gamma^{-p}$ for $p>0$, we have

[TABLE]

Proof 5.13**.**

Throughout this proof we consider probabilities of ellipsoids under the density $e^{-\gamma R_{\lambda}(\boldsymbol{w})}/Z$, and we abbreviate $R^{\star}_{\lambda,i}=R_{\lambda}(\boldsymbol{w}^{\star}_{\lambda,i})$. Applying Lemma 5.11 with $\boldsymbol{w}^{\star}_{\lambda}=\boldsymbol{w}^{\star}_{\lambda,i}$ readily gives

[TABLE]

This proves the first statement.

Now we look at the asymptotics of $\pi_{\gamma,r}(i)$ as $\gamma\to\infty$ assuming that $r=\gamma^{-p}$ for $p>0$. First, observe that for any $i\in\mathcal{I}$

[TABLE]

because $\lim_{\gamma\to\infty}L^{\star}_{i}(\gamma^{-p})=\mathcal{O}(1)$ and $\lambda_{\mathrm{min},i}+\lambda>0$. Thus, the numerator of (60) approaches $1$. Now, we turn our attention to the denominator. First, we consider global minimizers recalling our assumption that $R^{\star}_{\lambda,i}=0$. Denoting $\delta_{i,j}(\gamma)=e^{\gamma\left(R^{\star}_{\lambda,i}-R^{\star}_{\lambda,j}\right)}$, we observe that for all $\gamma\geq 0$ and $i\in\mathcal{I}^{\text{{glob}}}$,

[TABLE]

where the second case holds because the exponent in $\delta_{i,j}(\gamma)$ is negative. This implies

[TABLE]

Next, we consider the local minima, and observe that for all $\gamma\geq 0$ and $i\in\mathcal{I}^{\text{{subopt}}}$,

[TABLE]

Therefore, for all $i\in\mathcal{I}^{\text{{subopt}}}$, $\lim_{\gamma\rightarrow\infty}\pi_{\gamma,r}(i)=0$. This proves the second statement and completes the proof.

5.5 Global Excess Risk Bounds

We first show a nonasymptotic (i.e., finite $\gamma$) global excess risk bound.

Theorem 4.5 (restated)

Assume the same as in Lemma 4.2. Then for any $r\in[0,r_{0}]$ the global excess risk satisfies

[TABLE]

where the expectation is taken with respect to $I\sim\pi_{\gamma,r}$ and the probability of the complement of the minima is bounded as

[TABLE]

with $\alpha_{d/2}$ defined in (56).

Proof 5.14**.**

Denote the sample-dependent global excess risk by

[TABLE]

and let the probabilities \mathbb{P}\big{(}\mathcal{E}^{\star}_{i}(r)\big{)} and \mathbb{P}\big{(}\mathcal{C}^{\star}(r)\big{)} be defined with respect to the population Gibbs distribution $p_{\gamma}(\boldsymbol{w})\propto e^{-\gamma R_{\lambda}(\boldsymbol{w})}$ with $\gamma>0$.

We first focus on the first term on the right-hand side of $\Delta_{S}(\pi_{\gamma,r})$. By the law of total expectation, for any $r\in[0,r_{0}]$,

[TABLE]

An upper bound on \operatorname*{\mathbb{E}}\left[R(\boldsymbol{w})\,\big{|}\,\boldsymbol{w}\in\mathcal{E}^{\star}_{i}(r)\right] is given by Theorem 4.3, thus all that is left to show is that the probability of the complement is small. Since the ellipsoids are disjoint

[TABLE]

To upper bound \mathbb{P}\big{(}\mathcal{C}^{\star}(r)\big{)} we need a lower bound on \mathbb{P}\big{(}\mathcal{E}^{\star}(r)\big{)}. This is provided by the last inequality in Lemma 5.11, that is,

[TABLE]

where (63) is derived from the lower bound on the regularized Gamma function (55), and the last inequality is obtained from the Bernoulli inequality

[TABLE]

Thus,

[TABLE]

Taking expectation with respect to $S$, and combining Theorem 4.3 with (62) and Jensen’s inequality, we obtain

[TABLE]

The proof is concluded by stating the above with respect to radius $r\in[0,r_{0}]$ —recall that the radius cannot exceed $r_{0}$, the largest radius ensuring that ellipsoids remain disjoint.

Corollary 1 (restated)

Assume the same as in Lemma 4.2. Then, for any $r>0$, the global asymptotic pseudo-excess risk satisfies

[TABLE]

where $I$ is distributed according to

[TABLE]

Proof 5.15**.**

Recall that the global asymptotic pseudo-excess risk is defined as

[TABLE]

Distribution $\pi_{\infty}$ is given by Lemma 4.4, while the local excess risk centered at $\boldsymbol{w}^{\star}_{\lambda,I}$ is bounded by Theorem 4.3. This immediately yields the statement.

Acknowledgments

Authors would like to thank Olivier Bousquet, Sébastien Gerchinovitz, and Abbas Mehrabian for stimulating discussions on this work.