Sandwich Boosting for Accurate Estimation in Partially Linear Models for Grouped Data

TL;DR

This paper introduces a new 'sandwich loss' and a 'sandwich boosting' method for more accurate estimation in partially linear models with grouped data, especially when the covariance structure is unknown or misspecified.

Contribution

The paper proposes a novel sandwich loss function and a gradient boosting approach that improve linear parameter estimation in grouped data models, even under covariance misspecification.

Findings

Sandwich boosting outperforms traditional methods in simulations.

The approach achieves asymptotic Gaussianity and minimal variance.

Effective on both simulated and real-world datasets.

Abstract

We study partially linear models in settings where observations are arranged in independent groups but may exhibit within-group dependence. Existing approaches estimate linear model parameters through weighted least squares, with optimal weights (given by the inverse covariance of the response, conditional on the covariates) typically estimated by maximising a (restricted) likelihood from random effects modelling or by using generalised estimating equations. We introduce a new 'sandwich loss' whose population minimiser coincides with the weights of these approaches when the parametric forms for the conditional covariance are well-specified, but can yield arbitrarily large improvements in linear parameter estimation accuracy when they are not. Under relatively mild conditions, our estimated coefficients are asymptotically Gaussian and enjoy minimal variance among estimators with weights restricted to a given class of functions, when user-chosen regression methods are used to estimate nuisance functions. We further expand the class of functional forms for the weights that may be fitted beyond parametric models by leveraging the flexibility of modern machine learning methods within a new gradient boosting scheme for minimising the sandwich loss. We demonstrate the effectiveness of both the sandwich loss and what we call 'sandwich boosting' in a variety of settings with simulated and real-world data.