On the within-family Kullback-Leibler risk in Gaussian Predictive models

TL;DR

This paper analyzes the minimal Kullback-Leibler risk for Gaussian predictive densities in high-dimensional models, revealing sub-optimality in sparse cases and providing explicit bounds and optimal strategies for certain covariance structures.

Contribution

It introduces asymptotically sharp bounds and explicit formulas for predictive risk, connecting predictive density estimation with point estimation theory in high-dimensional Gaussian models.

Findings

Sub-family of Gaussian densities with data-dependent covariance achieves optimal predictive risk.

In sparse models, the class of all Gaussian densities is minimax sub-optimal.

Explicit risk bounds are derived for specific covariance structures.

Abstract

We consider estimating the predictive density under Kullback-Leibler loss in a high-dimensional Gaussian model. Decision theoretic properties of the within-family prediction error -- the minimal risk among estimates in the class $\mathcal{G}$ of all Gaussian densities are discussed. We show that in sparse models, the class $\mathcal{G}$ is minimax sub-optimal. We produce asymptotically sharp upper and lower bounds on the within-family prediction errors for various subfamilies of $\mathcal{G}$. Under mild regularity conditions, in the sub-family where the covariance structure is represented by a single data dependent parameter $\Shat=\dhat \cdot I$, the Kullback-Leiber risk has a tractable decomposition which can be subsequently minimized to yield optimally flattened predictive density estimates. The optimal predictive risk can be explicitly expressed in terms of the corresponding mean square error of the location estimate, and so, the role of shrinkage in the predictive regime can be determined based on point estimation theory results. Our results demonstrate that some of the decision theoretic parallels between predictive density estimation and point estimation regimes can be explained by second moment based concentration properties of the quadratic loss.