Validation design I: construction of validation designs via kernel herding

TL;DR

This paper introduces a novel validation design method using kernel herding to construct nested validation sets that effectively estimate prediction error, emphasizing space-filling and weighted properties for improved accuracy.

Contribution

It proposes a new validation design construction technique based on kernel herding that outperforms traditional methods in estimating prediction error.

Findings

Kernel herding-based validation designs outperform random and low-discrepancy designs.

Weighted validation points improve error estimation accuracy.

Numerical experiments confirm the method's superiority in various settings.

Abstract

We construct validation designs $Z_m$ aimed at estimating the integrated squared prediction error of a given design $X_n$. Our approach is based on the minimization of a maximum mean discrepancy for a particular kernel, conditional on $X_n$, so that sequences of nested validation designs can be constructed incrementally by kernel herding. Numerical experiments show that key features for a good validation design are its space-filling properties, in order to fill the holes left by $X_n$ and properly explore the whole design space, and the suitable weighting of its points, since evaluations far from $X_n$ tend to overestimate the global error. A dedicated weighting method, based on a particular kernel, is proposed. Numerical simulations with random functions show the superiority the method over more traditional validation based on random designs, low-discrepancy sequences, or leave-one-out cross validation.