The Contextual Lasso: Sparse Linear Models via Deep Neural Networks

TL;DR

This paper introduces the contextual lasso, a method that combines deep neural networks with sparse linear models to produce interpretable, context-dependent models that are as predictive as deep networks but more transparent.

Contribution

It proposes a novel estimator that learns sparse linear models with coefficients varying by contextual features using deep neural networks and a new lasso regularizer.

Findings

Models are sparser than regular lasso without losing predictive accuracy.

The method maintains high interpretability while matching deep neural network performance.

Experiments on real and synthetic data validate the approach's effectiveness.

Abstract

Sparse linear models are one of several core tools for interpretable machine learning, a field of emerging importance as predictive models permeate decision-making in many domains. Unfortunately, sparse linear models are far less flexible as functions of their input features than black-box models like deep neural networks. With this capability gap in mind, we study a not-uncommon situation where the input features dichotomize into two groups: explanatory features, which are candidates for inclusion as variables in an interpretable model, and contextual features, which select from the candidate variables and determine their effects. This dichotomy leads us to the contextual lasso, a new statistical estimator that fits a sparse linear model to the explanatory features such that the sparsity pattern and coefficients vary as a function of the contextual features. The fitting process learns this function nonparametrically via a deep neural network. To attain sparse coefficients, we train the network with a novel lasso regularizer in the form of a projection layer that maps the network's output onto the space of $\ell_1$-constrained linear models. An extensive suite of experiments on real and synthetic data suggests that the learned models, which remain highly transparent, can be sparser than the regular lasso without sacrificing the predictive power of a standard deep neural network.