Optimal Kernel Quantile Learning with Random Features

TL;DR

This paper develops a theoretical framework for kernel quantile regression with random features, providing optimal learning rates and extending applicability to various loss functions and real-world data.

Contribution

It introduces a novel analysis of KQR-RF, establishing minimax optimal learning rates and extending the method to broader settings with Lipschitz losses and agnostic data.

Findings

Capacity-dependent learning rates are established for KQR-RF.

Theoretical results are validated through simulations and real data experiments.

The analysis extends to Lipschitz continuous losses and agnostic settings.

Abstract

The random feature (RF) approach is a well-established and efficient tool for scalable kernel methods, but existing literature has primarily focused on kernel ridge regression with random features (KRR-RF), which has limitations in handling heterogeneous data with heavy-tailed noises. This paper presents a generalization study of kernel quantile regression with random features (KQR-RF), which accounts for the non-smoothness of the check loss in KQR-RF by introducing a refined error decomposition and establishing a novel connection between KQR-RF and KRR-RF. Our study establishes the capacity-dependent learning rates for KQR-RF under mild conditions on the number of RFs, which are minimax optimal up to some logarithmic factors. Importantly, our theoretical results, utilizing a data-dependent sampling strategy, can be extended to cover the agnostic setting where the target quantile function may not precisely align with the assumed kernel space. By slightly modifying our assumptions, the capacity-dependent error analysis can also be applied to cases with Lipschitz continuous losses, enabling broader applications in the machine learning community. To validate our theoretical findings, simulated experiments and a real data application are conducted.