Stochastic Gradients under Nuisances

TL;DR

This paper analyzes stochastic gradient algorithms for learning problems with unknown nuisance parameters, providing convergence guarantees and showing that orthogonalization techniques can ensure reliable optimization.

Contribution

It establishes non-asymptotic convergence results for stochastic gradients in the presence of nuisances and introduces variants with approximate orthogonalization for broader applicability.

Findings

Classical stochastic gradient methods can converge despite nuisances under Neyman orthogonality.

Orthogonalized update algorithms achieve similar convergence rates without strict orthogonality.

Applications include orthogonal statistical learning, double machine learning, and causal inference.

Abstract

Stochastic gradient optimization is the dominant learning paradigm for a variety of scenarios, from classical supervised learning to modern self-supervised learning. We consider stochastic gradient algorithms for learning problems whose objectives rely on unknown nuisance parameters, and establish non-asymptotic convergence guarantees. Our results show that, while the presence of a nuisance can alter the optimum and upset the optimization trajectory, the classical stochastic gradient algorithm may still converge under appropriate conditions, such as Neyman orthogonality. Moreover, even when Neyman orthogonality is not satisfied, we show that an algorithm variant with approximately orthogonalized updates (with an approximately orthogonalized gradient oracle) may achieve similar convergence rates. Examples from orthogonal statistical learning/double machine learning and causal inference are discussed.