Semi-Supervised Deep Sobolev Regression: Estimation and Variable Selection by ReQU Neural Network

TL;DR

This paper introduces SDORE, a semi-supervised deep learning method using ReQU neural networks for nonparametric regression and gradient estimation, with theoretical guarantees and practical validation.

Contribution

It presents the first provable neural network approach for simultaneous regression and gradient estimation, leveraging unlabeled data for improved accuracy.

Findings

Achieves minimax optimal convergence rates in $L^{2}$-norm.

Provides a convergence rate for the plug-in gradient estimator under domain shift.

Demonstrates effectiveness through extensive numerical simulations.

Abstract

We propose SDORE, a Semi-supervised Deep Sobolev Regressor, for the nonparametric estimation of the underlying regression function and its gradient. SDORE employs deep ReQU neural networks to minimize the empirical risk with gradient norm regularization, allowing the approximation of the regularization term by unlabeled data. Our study includes a thorough analysis of the convergence rates of SDORE in $L^{2}$-norm, achieving the minimax optimality. Further, we establish a convergence rate for the associated plug-in gradient estimator, even in the presence of significant domain shift. These theoretical findings offer valuable insights for selecting regularization parameters and determining the size of the neural network, while showcasing the provable advantage of leveraging unlabeled data in semi-supervised learning. To the best of our knowledge, SDORE is the first provable neural network-based approach that simultaneously estimates the regression function and its gradient, with diverse applications such as nonparametric variable selection. The effectiveness of SDORE is validated through an extensive range of numerical simulations.