Integrating Complex Covariate Transformations in Generalized Additive Models

TL;DR

This paper introduces a novel framework for embedding complex, interpretable covariate transformations directly into generalized additive models, enhancing feature engineering within the modeling process.

Contribution

It develops a flexible, joint estimation approach for covariate transformations in GAMs, supported by efficient computation and Bayesian uncertainty quantification.

Findings

Applied to electricity demand forecasting in Great Britain.

Modeled house prices in London with complex covariate transformations.

Provided an R package 'gamFactory' for practical implementation.

Abstract

Transformations of covariates are widely used in applied statistics to improve interpretability and to satisfy assumptions required for valid inference. More broadly, feature engineering encompasses a wider set of practices aimed at enhancing predictive performance, and is typically performed as part of a data pre-processing step. In contrast, this paper integrates a substantial component of the feature engineering process directly into the modelling stage. This is achieved by introducing a novel general framework for embedding interpretable covariate transformations within multi-parameter Generalised Additive Models (GAMs). Our framework accommodates any sufficiently differentiable scalar-valued transformation of potentially high-dimensional and complex covariates. These transformations are treated as integral model components, with their parameters estimated jointly with regression coefficients via maximum a posteriori (MAP) methods, and joint uncertainty quantified via approximate Bayesian techniques. Smoothing parameters are selected in an empirical Bayes framework using a Laplace approximation to the marginal likelihood, supported by efficient computation based on implicit differentiation methods. We demonstrate the flexibility and practical value of the proposed methodology through applications to forecasting electricity net-demand in Great Britain and to modelling house prices in London. Methods for building and fitting GAMs with nested transformations are provided by the gamFactory R package, available at https://github.com/mfasiolo/gamFactory, while the code for reproducing the results in this paper is available at https://doi.org/10.5281/zenodo.19239350.