Regularization Strategies for Quantile Regression

TL;DR

This paper explores various regularization techniques for quantile regression, including expected pinball loss, deep lattice networks for non-crossing quantiles, and rate constraints, demonstrating improved calibration and fairness on multiple datasets.

Contribution

It introduces novel regularization methods for quantile regression, such as using deep lattice networks and rate constraints, to improve prediction accuracy, calibration, and fairness.

Findings

Expected pinball loss regularizes quantile predictions effectively.

Deep lattice networks prevent quantile crossing by enforcing monotonicity.

Rate constraints enhance calibration and fairness in quantile predictions.

Abstract

We investigate different methods for regularizing quantile regression when predicting either a subset of quantiles or the full inverse CDF. We show that minimizing an expected pinball loss over a continuous distribution of quantiles is a good regularizer even when only predicting a specific quantile. For predicting multiple quantiles, we propose achieving the classic goal of non-crossing quantiles by using deep lattice networks that treat the quantile as a monotonic input feature, and we discuss why monotonicity on other features is an apt regularizer for quantile regression. We show that lattice models enable regularizing the predicted distribution to a location-scale family. Lastly, we propose applying rate constraints to improve the calibration of the quantile predictions on specific subsets of interest and improve fairness metrics. We demonstrate our contributions on simulations, benchmark datasets, and real quantile regression problems.