LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models

TL;DR

LIT-LVM introduces a structured regularization method using latent variable models to improve the estimation of interaction coefficients in linear predictors, especially in high-dimensional data, enhancing prediction accuracy and interpretability.

Contribution

The paper proposes LIT-LVM, a novel low-dimensional latent variable approach for regularizing interaction terms in linear models, outperforming existing methods in accuracy and feature analysis.

Findings

LIT-LVM outperforms elastic net, hierarchical lasso, and factorization machines in prediction accuracy.

LIT-LVM effectively handles high-dimensional interaction modeling.

Provides interpretable low-dimensional feature representations.

Abstract

Some of the simplest, yet most frequently used predictors in statistics and machine learning use weighted linear combinations of features. Such linear predictors can model non-linear relationships between features by adding interaction terms corresponding to the products of all pairs of features. We consider the problem of accurately estimating coefficients for interaction terms in linear predictors. We hypothesize that the coefficients for different interaction terms have an approximate low-dimensional structure and represent each feature by a latent vector in a low-dimensional space. This low-dimensional representation can be viewed as a structured regularization approach that further mitigates overfitting in high-dimensional settings beyond standard regularizers such as the lasso and elastic net. We demonstrate that our approach, called LIT-LVM, achieves superior prediction accuracy compared to the elastic net, hierarchical lasso, and factorization machines on a wide variety of simulated and real data, particularly when the number of interaction terms is high compared to the number of samples. LIT-LVM also provides low-dimensional latent representations for features that are useful for visualizing and analyzing their relationships.