Semi-parametric Bayesian inference under Neyman orthogonality

TL;DR

This paper explores Bayesian inference in semi-parametric models with Neyman orthogonality, demonstrating that a two-step procedure with plug-in nuisance parameters can yield valid marginal posteriors with good frequentist properties, especially in causal inference.

Contribution

It introduces a Bayesian approach using Dirichlet process and bootstrap for semi-parametric models satisfying Neyman orthogonality, ensuring valid inference despite nuisance parameter estimation.

Findings

Posterior of the targeted parameter shows good frequentist properties.

Plug-in of nuisance parameters has negligible effect on marginal inference.

Asymptotic invariance of the posterior even without Neyman orthogonality.

Abstract

The validity of two-step or plug-in inference methods is questioned in the Bayesian framework. We study semi-parametric models where the plug-in of a non-parametrically modelled nuisance component is used. We show that when the nuisance and targeted parameters satisfy a Neyman orthogonal score property, the approach of cutting feedback through a two-step procedure is a valid way of conducting Bayesian inference. Our method relies on a non-parametric Bayesian formulation based on the Dirichlet process and the Bayesian bootstrap. We show that the marginal posterior of the targeted parameter exhibits good frequentist properties despite not accounting for the inferential uncertainty of the nuisance parameter. We adopt this approach in Bayesian causal inference problems where the nuisance propensity score model is estimated to obtain marginal inference for the treatment effect parameter, and demonstrate that a plug-in of the propensity score has a negligible effect on marginal posterior inference for the causal contrast. We investigate the absence of Neyman orthogonality and exploit our findings to show that in conventional two-step procedures, the posterior distribution converges under weaker restrictions than those needed in the frequentist sequel. For a simple family of useful scores, we demonstrate that even in the absence of Neyman orthogonality, the posterior distribution is asymptotically unchanged by the estimation of the nuisance parameter, merely provided the latter estimator is consistent.