Minimax Linear Regression under the Quantile Risk

TL;DR

This paper develops minimax procedures for linear regression under quantile risk, establishing optimal bounds and demonstrating the minimaxity of certain estimators in Gaussian noise settings.

Contribution

It provides the first exact minimax quantile risk characterization for linear regression with Gaussian noise and extends minimaxity results to various error functions.

Findings

Exact minimax quantile risk derived for Gaussian noise setting.

Minimaxity of OLS established for square error.

Extension of results to p-th power errors for p in (2, ∞).

Abstract

We study the problem of designing minimax procedures in linear regression under the quantile risk. We start by considering the realizable setting with independent Gaussian noise, where for any given noise level and distribution of inputs, we obtain the exact minimax quantile risk for a rich family of error functions and establish the minimaxity of OLS. This improves on the known lower bounds for the special case of square error, and provides us with a lower bound on the minimax quantile risk over larger sets of distributions. Under the square error and a fourth moment assumption on the distribution of inputs, we show that this lower bound is tight over a larger class of problems. Specifically, we prove a matching upper bound on the worst-case quantile risk of a variant of the recently proposed min-max regression procedure, thereby establishing its minimaxity, up to absolute constants. We illustrate the usefulness of our approach by extending this result to all $p$-th power error functions for $p \in (2, \infty)$. Along the way, we develop a generic analogue to the classical Bayesian method for lower bounding the minimax risk when working with the quantile risk, as well as a tight characterization of the quantiles of the smallest eigenvalue of the sample covariance matrix.