Generalized Kernel Regularized Least Squares

TL;DR

This paper introduces generalized KRLS (gKRLS), a flexible and computationally efficient extension of Kernel Regularized Least Squares that allows for complex model components and scales to large datasets.

Contribution

The paper reformulates KRLS as a hierarchical model, enabling integration with other model components and introduces random sketching for faster estimation.

Findings

gKRLS can be fitted on datasets with tens of thousands of observations in under one minute.

gKRLS allows combining KRLS with random effects, splines, and fixed effects.

State-of-the-art models requiring multiple fits can be estimated quickly using gKRLS.

Abstract

Kernel Regularized Least Squares (KRLS) is a popular method for flexibly estimating models that may have complex relationships between variables. However, its usefulness to many researchers is limited for two reasons. First, existing approaches are inflexible and do not allow KRLS to be combined with theoretically-motivated extensions such as random effects, unregularized fixed effects, or non-Gaussian outcomes. Second, estimation is extremely computationally intensive for even modestly sized datasets. Our paper addresses both concerns by introducing generalized KRLS (gKRLS). We note that KRLS can be re-formulated as a hierarchical model thereby allowing easy inference and modular model construction where KRLS can be used alongside random effects, splines, and unregularized fixed effects. Computationally, we also implement random sketching to dramatically accelerate estimation while incurring a limited penalty in estimation quality. We demonstrate that gKRLS can be fit on datasets with tens of thousands of observations in under one minute. Further, state-of-the-art techniques that require fitting the model over a dozen times (e.g. meta-learners) can be estimated quickly.