How Many Iterations are Sufficient for Semiparametric Estimation?

TL;DR

This paper derives a theoretical formula for the minimal number of iterations needed in semiparametric estimation to achieve efficiency, highlighting how iteration count relates to convergence rates and model regularization.

Contribution

It provides the first theoretical formula for the minimal iteration count in semiparametric estimation, applicable across various regularizations and high-dimensional settings.

Findings

Minimal iteration count depends on initial and nuisance estimate convergence rates.

More than the minimal iterations, additional steps improve higher order efficiency.

Minimal iterations suffice for recovering sparsity in high-dimensional models.

Abstract

A common practice in obtaining a semiparametric efficient estimate is through iteratively maximizing the (penalized) log-likelihood w.r.t. its Euclidean parameter and functional nuisance parameter via Newton-Raphson algorithm. The purpose of this paper is to provide a formula in calculating the minimal number of iterations $k^\ast$ needed to produce an efficient estimate $\hat\theta_n^{(k^\ast)}$ from a theoretical point of view. We discover that (a) $k^\ast$ depends on the convergence rates of the initial estimate and nuisance estimate; (b) more than $k^\ast$ iterations, i.e., $k$, will only improve the higher order asymptotic efficiency of $\hat\theta_n^{(k)}$; (c) $k^\ast$ iterations are also sufficient for recovering the estimation sparsity in high dimensional data. These general conclusions hold, in particular, when the nuisance parameter is not estimable at root-n rate, and apply to semiparametric models estimated under various regularizations, e.g., kernel or penalized estimation. This paper provides a first general theoretical justification for the "one-/two-step iteration" phenomena observed in the literature, and may be useful in reducing the bootstrap computational cost for the semiparametric models.