The Conjugate Domain Dichotomy: Exact Risk of M-Estimators under Infinite-Variance Noise in High Dimensions

TL;DR

This paper analyzes the asymptotic risk of high-dimensional M-estimators with infinite-variance noise, revealing how the loss function's conjugate domain influences estimator behavior and risk bounds.

Contribution

It provides a precise characterization of the risk behavior of M-estimators under infinite-variance noise, highlighting the role of the loss function's conjugate domain in high dimensions.

Findings

Bounded conjugate domain leads to bounded risk without external info.

Unbounded conjugate domain results in risk divergence or need for transfer regularization.

Exact asymptotic risk for squared loss derived using Convex Gaussian Minimax Theorem.

Abstract

This paper studies high-dimensional M-estimation in the proportional asymptotic regime (p/n -> gamma > 0) when the noise distribution has infinite variance. For noise with regularly-varying tails of index alpha in (1,2), we establish that the asymptotic behavior of a regularized M-estimator is governed by a single geometric property of the loss function: the boundedness of the domain of its Fenchel conjugate. When this conjugate domain is bounded -- as is the case for the Huber, absolute-value, and quantile loss functions -- the dual variable in the min-max formulation of the estimator is confined, the effective noise reduces to the finite first absolute moment of the noise distribution, and the estimator achieves bounded risk without recourse to external information. When the conjugate domain is unbounded -- as for the squared loss -- the dual variable scales with the noise, the effective noise involves the diverging second moment, and bounded risk can be achieved only through transfer regularization toward an external prior. For the squared-loss class specifically, we derive the exact asymptotic risk via the Convex Gaussian Minimax Theorem under a noise-adapted regularization scaling. The resulting risk converges to a universal floor that is independent of the regularizer, yielding a loss-risk trichotomy: squared-loss estimators without transfer diverge; Huber-loss estimators achieve bounded but non-vanishing risk; transfer-regularized estimators attain the floor.