Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and its Asymptotic Optimality

TL;DR

This paper introduces a scalable, data-driven method called distributed GCV for tuning kernel ridge regression in large datasets, and proves its asymptotic optimality within the divide-and-conquer framework.

Contribution

It proposes a modified GCV score for d-KRR that is computationally scalable and asymptotically optimal, extending GCV's optimality to divide-and-conquer methods.

Findings

dGCV is computationally scalable for massive datasets.

Minimizing dGCV is asymptotically equivalent to minimizing the true loss.

The method extends GCV optimality results to divide-and-conquer kernel ridge regression.

Abstract

Tuning parameter selection is of critical importance for kernel ridge regression. To this date, data driven tuning method for divide-and-conquer kernel ridge regression (d-KRR) has been lacking in the literature, which limits the applicability of d-KRR for large data sets. In this paper, by modifying the Generalized Cross-validation (GCV, Wahba, 1990) score, we propose a distributed Generalized Cross-Validation (dGCV) as a data-driven tool for selecting the tuning parameters in d-KRR. Not only the proposed dGCV is computationally scalable for massive data sets, it is also shown, under mild conditions, to be asymptotically optimal in the sense that minimizing the dGCV score is equivalent to minimizing the true global conditional empirical loss of the averaged function estimator, extending the existing optimality results of GCV to the divide-and-conquer framework.