Transfer Learning with General Estimating Equations

TL;DR

This paper introduces a semiparametric efficient inference method for transfer learning with covariate shift, utilizing orthogonal estimating equations and machine learning for nuisance function estimation, demonstrated through simulations and ozone pollution data.

Contribution

It develops a novel orthogonal estimation framework for transfer learning that avoids density ratio and regression errors, achieving semiparametric efficiency.

Findings

Estimator attains the semiparametric efficiency bound.

Inference is valid via Wilks' theorem without sample splitting.

Method performs well in simulations and ozone pollution study.

Abstract

We consider statistical inference for parameters defined by general estimating equations under the covariate shift transfer learning. Different from the commonly used density ratio weighting approach, we undertake a set of formulations to make the statistical inference semiparametric efficient with simple inference. It starts with re-constructing the estimation equations to make them Neyman orthogonal, which facilitates more robustness against errors in the estimation of two key nuisance functions, the density ratio and the conditional mean of the moment function. We present a divergence-based method to estimate the density ratio function, which is amenable to machine learning algorithms including the deep learning. To address the challenge that the conditional mean is parametric-dependent, we adopt a nonparametric multiple-imputation strategy that avoids regression at all possible parameter values. With the estimated nuisance functions and the orthogonal estimation equation, the inference for the target parameter is formulated via the empirical likelihood without sample splittings. We show that the proposed estimator attains the semiparametric efficiency bound, and the inference can be conducted with the Wilks' theorem. The proposed method is further evaluated by simulations and an empirical study on a transfer learning inference for ground-level ozone pollution