Linked shrinkage to improve estimation of interaction effects in regression models

TL;DR

This paper introduces a novel local shrinkage model for regression with interaction effects, improving estimation accuracy and inference in high-dimensional settings, and providing efficient variable importance assessment.

Contribution

It proposes a soft linking approach between main effects and interactions using local shrinkage, enhancing estimation and inference in complex regression models.

Findings

Borrowing strength between effects improves estimation accuracy.

The method provides reliable variable importance scores with uncertainty quantification.

Models are competitive with random forests in large datasets.

Abstract

We address a classical problem in statistics: adding two-way interaction terms to a regression model. As the covariate dimension increases quadratically, we develop an estimator that adapts well to this increase, while providing accurate estimates and appropriate inference. Existing strategies overcome the dimensionality problem by only allowing interactions between relevant main effects. Building on this philosophy, we implement a softer link between the two types of effects using a local shrinkage model. We empirically show that borrowing strength between the amount of shrinkage for main effects and their interactions can strongly improve estimation of the regression coefficients. Moreover, we evaluate the potential of the model for inference, which is notoriously hard for selection strategies. Large-scale cohort data are used to provide realistic illustrations and evaluations. Comparisons with other methods are provided. The evaluation of variable importance is not trivial in regression models with many interaction terms. Therefore, we derive a new analytical formula for the Shapley value, which enables rapid assessment of individual-specific variable importance scores and their uncertainties. Finally, while not targeting for prediction, we do show that our models can be very competitive to a more advanced machine learner, like random forest, even for fairly large sample sizes. The implementation of our method in RStan is fairly straightforward, allowing for adjustments to specific needs.