Estimating the minimizer and the minimum value of a regression function under passive design

TL;DR

This paper introduces a new method for accurately estimating the minimizer and minimum value of a smooth, strongly convex regression function from noisy observations, using a combination of projected gradient descent and nonparametric estimation.

Contribution

The paper develops a novel two-stage estimation procedure with non-asymptotic bounds, achieving minimax optimal rates for strongly convex functions.

Findings

Estimates the minimizer with quadratic risk bounds.

Provides minimax lower bounds matching the upper bounds.

Achieves optimal convergence rates for both minimizer and minimum value estimation.

Abstract

We propose a new method for estimating the minimizer $\boldsymbol{x}^*$ and the minimum value $f^*$ of a smooth and strongly convex regression function $f$ from the observations contaminated by random noise. Our estimator $\boldsymbol{z}_n$ of the minimizer $\boldsymbol{x}^*$ is based on a version of the projected gradient descent with the gradient estimated by a regularized local polynomial algorithm. Next, we propose a two-stage procedure for estimation of the minimum value $f^*$ of regression function $f$. At the first stage, we construct an accurate enough estimator of $\boldsymbol{x}^*$, which can be, for example, $\boldsymbol{z}_n$. At the second stage, we estimate the function value at the point obtained in the first stage using a rate optimal nonparametric procedure. We derive non-asymptotic upper bounds for the quadratic risk and optimization error of $\boldsymbol{z}_n$, and for the risk of estimating $f^*$. We establish minimax lower bounds showing that, under certain choice of parameters, the proposed algorithms achieve the minimax optimal rates of convergence on the class of smooth and strongly convex functions.