Colorful Pinball: Density-Weighted Quantile Regression for Conditional Guarantee of Conformal Prediction

TL;DR

This paper introduces a density-weighted pinball loss for quantile regression to improve conditional coverage guarantees in conformal prediction, supported by theoretical analysis and empirical validation.

Contribution

It proposes a novel density-weighted quantile regression method with a three-headed network and provides non-asymptotic guarantees for enhanced conditional coverage.

Findings

Significant improvement in conditional coverage on real-world datasets.

Theoretical non-asymptotic guarantees for the proposed method.

Effective estimation of density weights via finite differences.

Abstract

Although conformal prediction provides robust marginal coverage guarantees, achieving reliable conditional coverage for specific inputs remains challenging. While exact distribution-free conditional coverage is impossible with finite samples, recent work has focused on improving the conditional coverage of standard conformal procedures. Distinct from approaches that target relaxed notions of conditional coverage, we directly target the mean squared error of conditional coverage by refining the quantile regression components that underpin many conformal methods. Leveraging a Taylor expansion, we derive a sharp surrogate objective for quantile regression: a density-weighted pinball loss, where the weights are given by the conditional density of the nonconformity score evaluated at the true quantile. We propose a three-headed quantile network that estimates these weights via finite differences using auxiliary quantile levels at $1-\alpha \pm \delta$, subsequently fine-tuning the central quantile by optimizing the weighted loss. We provide a theoretical analysis with exact non-asymptotic guarantees characterizing the resulting excess risk. Extensive experiments on diverse high-dimensional real-world datasets demonstrate remarkable improvements in conditional coverage performance.