A sparse PAC-Bayesian approach for high-dimensional quantile prediction

TL;DR

This paper introduces a novel sparse PAC-Bayesian method for high-dimensional quantile prediction, combining pseudo-Bayesian modeling with Langevin Monte Carlo, and providing strong theoretical guarantees and competitive empirical results.

Contribution

It develops a new probabilistic approach using a scaled Student-t prior and PAC-Bayes bounds for high-dimensional quantile regression, addressing limitations of existing Bayesian methods.

Findings

Achieves minimax-optimal prediction error.

Provides non-asymptotic oracle inequalities.

Performs competitively on real-world data.

Abstract

Quantile regression, a robust method for estimating conditional quantiles, has advanced significantly in fields such as econometrics, statistics, and machine learning. In high-dimensional settings, where the number of covariates exceeds sample size, penalized methods like lasso have been developed to address sparsity challenges. Bayesian methods, initially connected to quantile regression via the asymmetric Laplace likelihood, have also evolved, though issues with posterior variance have led to new approaches, including pseudo/score likelihoods. This paper presents a novel probabilistic machine learning approach for high-dimensional quantile prediction. It uses a pseudo-Bayesian framework with a scaled Student-t prior and Langevin Monte Carlo for efficient computation. The method demonstrates strong theoretical guarantees, through PAC-Bayes bounds, that establish non-asymptotic oracle inequalities, showing minimax-optimal prediction error and adaptability to unknown sparsity. Its effectiveness is validated through simulations and real-world data, where it performs competitively against established frequentist and Bayesian techniques.