Lifting Interpretability-Performance Trade-off via Automated Feature Engineering

TL;DR

This paper introduces a method to develop interpretable models with high accuracy by leveraging surrogate black-box models for feature engineering, validated through extensive benchmarking on tabular datasets.

Contribution

It proposes a novel approach that uses elastic black-box models to automate feature engineering, balancing interpretability and performance without manual feature crafting.

Findings

Extracting information from complex models can enhance linear model performance.

Complex models do not always outperform simpler linear models.

Automated feature engineering can improve interpretability without sacrificing accuracy.

Abstract

Complex black-box predictive models may have high performance, but lack of interpretability causes problems like lack of trust, lack of stability, sensitivity to concept drift. On the other hand, achieving satisfactory accuracy of interpretable models require more time-consuming work related to feature engineering. Can we train interpretable and accurate models, without timeless feature engineering? We propose a method that uses elastic black-boxes as surrogate models to create a simpler, less opaque, yet still accurate and interpretable glass-box models. New models are created on newly engineered features extracted with the help of a surrogate model. We supply the analysis by a large-scale benchmark on several tabular data sets from the OpenML database. There are two results 1) extracting information from complex models may improve the performance of linear models, 2) questioning a common myth that complex machine learning models outperform linear models.