Ensemble Kernel Methods, Implicit Regularization and Determinantal Point Processes

TL;DR

This paper explores how determinantal point processes (DPPs) induce implicit regularization in kernel regression and demonstrates that ensemble methods using DPPs can improve performance on datasets with redundant information.

Contribution

It provides a theoretical link between DPP sampling and implicit regularization in ridgeless kernel regression, and proposes ensemble approaches leveraging DPPs for better results.

Findings

DPP sampling induces implicit regularization in ridgeless kernel regression.

Ensemble of ridgeless regressors benefits datasets with redundant information.

Initial empirical results show promising improvements using DPP-based ensembles.

Abstract

By using the framework of Determinantal Point Processes (DPPs), some theoretical results concerning the interplay between diversity and regularization can be obtained. In this paper we show that sampling subsets with kDPPs results in implicit regularization in the context of ridgeless Kernel Regression. Furthermore, we leverage the common setup of state-of-the-art DPP algorithms to sample multiple small subsets and use them in an ensemble of ridgeless regressions. Our first empirical results indicate that ensemble of ridgeless regressors can be interesting to use for datasets including redundant information.