Optimal Sampling Density for Nonparametric Regression

TL;DR

This paper introduces a robust, model-agnostic active learning strategy for nonparametric regression that derives an optimal sampling density to minimize generalization error, validated through simulations.

Contribution

It provides a novel, closed-form optimal sampling density for local polynomial smoothing that accounts for local complexity, noise, and test density, enhancing active learning.

Findings

Outperforms existing model-agnostic active learning methods in simulations.

Derives a closed-form expression for optimal training density.

Demonstrates robustness and interpretability of the proposed approach.

Abstract

We propose a novel active learning strategy for regression, which is model-agnostic, robust against model mismatch, and interpretable. Assuming that a small number of initial samples are available, we derive the optimal training density that minimizes the generalization error of local polynomial smoothing (LPS) with its kernel bandwidth tuned locally: We adopt the mean integrated squared error (MISE) as a generalization criterion, and use the asymptotic behavior of the MISE as well as the locally optimal bandwidths (LOB) - the bandwidth function that minimizes MISE in the asymptotic limit. The asymptotic expression of our objective then reveals the dependence of the MISE on the training density, enabling analytic minimization. As a result,we obtain the optimal training density in a closed-form. The almost model-free nature of our approach thus helps to encode the essential properties of the target problem, providing a robust and model-agnostic active learning strategy. Furthermore, the obtained training density factorizes the influence of local function complexity, noise level and test density in a transparent and interpretable way. We validate our theory in numerical simulations, and show that the proposed active learning method outperforms the existing state-of-the-art model-agnostic approaches.