Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models

TL;DR

This paper introduces a multi-objective hyperparameter optimization framework for tabular supervised learning models that balances predictive accuracy and interpretability, using a novel evolutionary algorithm to generate diverse, high-quality models.

Contribution

It proposes a model-agnostic, multi-objective optimization approach that incorporates interpretability constraints into hyperparameter tuning for tabular data models.

Findings

The framework effectively finds Pareto optimal models balancing performance and interpretability.

It outperforms state-of-the-art models like XGBoost and Explainable Boosting Machine in benchmark tests.

The evolutionary algorithm efficiently explores the augmented search space of feature groups and constraints.

Abstract

We present a model-agnostic framework for jointly optimizing the predictive performance and interpretability of supervised machine learning models for tabular data. Interpretability is quantified via three measures: feature sparsity, interaction sparsity of features, and sparsity of non-monotone feature effects. By treating hyperparameter optimization of a machine learning algorithm as a multi-objective optimization problem, our framework allows for generating diverse models that trade off high performance and ease of interpretability in a single optimization run. Efficient optimization is achieved via augmentation of the search space of the learning algorithm by incorporating feature selection, interaction and monotonicity constraints into the hyperparameter search space. We demonstrate that the optimization problem effectively translates to finding the Pareto optimal set of groups of selected features that are allowed to interact in a model, along with finding their optimal monotonicity constraints and optimal hyperparameters of the learning algorithm itself. We then introduce a novel evolutionary algorithm that can operate efficiently on this augmented search space. In benchmark experiments, we show that our framework is capable of finding diverse models that are highly competitive or outperform state-of-the-art XGBoost or Explainable Boosting Machine models, both with respect to performance and interpretability.