Linear Iterative Feature Embedding: An Ensemble Framework for Interpretable Model

TL;DR

LIFE is an ensemble framework that combines multiple narrow neural networks trained on data subsets to produce an interpretable, accurate, and computationally efficient model, outperforming several benchmarks.

Contribution

The paper introduces LIFE, a novel ensemble method that fits wide single-hidden-layer neural networks through a three-step process, enhancing interpretability and efficiency.

Findings

LIFE outperforms single-hidden-layer NNs, FFNN, Xgboost, and RF in accuracy.

LIFE provides easily visualizable variable importance and interaction effects.

LIFE is computationally efficient due to parallel training of base learners.

Abstract

A new ensemble framework for interpretable model called Linear Iterative Feature Embedding (LIFE) has been developed to achieve high prediction accuracy, easy interpretation and efficient computation simultaneously. The LIFE algorithm is able to fit a wide single-hidden-layer neural network (NN) accurately with three steps: defining the subsets of a dataset by the linear projections of neural nodes, creating the features from multiple narrow single-hidden-layer NNs trained on the different subsets of the data, combining the features with a linear model. The theoretical rationale behind LIFE is also provided by the connection to the loss ambiguity decomposition of stack ensemble methods. Both simulation and empirical experiments confirm that LIFE consistently outperforms directly trained single-hidden-layer NNs and also outperforms many other benchmark models, including multi-layers Feed Forward Neural Network (FFNN), Xgboost, and Random Forest (RF) in many experiments. As a wide single-hidden-layer NN, LIFE is intrinsically interpretable. Meanwhile, both variable importance and global main and interaction effects can be easily created and visualized. In addition, the parallel nature of the base learner building makes LIFE computationally efficient by leveraging parallel computing.