What makes an Ensemble (Un) Interpretable?

TL;DR

This paper investigates the interpretability of ensemble models using computational complexity theory, revealing how factors like number and type of base models affect their interpretability and intractability.

Contribution

It provides a rigorous mathematical analysis of ensemble interpretability, highlighting complexity patterns influenced by ensemble size and base model type.

Findings

Small ensembles of decision trees are efficiently interpretable.

Interpreting ensembles with linear models remains intractable.

Interpretability complexity is influenced by ensemble size and base model type.

Abstract

Ensemble models are widely recognized in the ML community for their limited interpretability. For instance, while a single decision tree is considered interpretable, ensembles of trees (e.g., boosted trees) are often treated as black-boxes. Despite this folklore recognition, there remains a lack of rigorous mathematical understanding of what particularly makes an ensemble (un)-interpretable, including how fundamental factors like the (1) *number*, (2) *size*, and (3) *type* of base models influence its interpretability. In this work, we seek to bridge this gap by applying concepts from computational complexity theory to study the challenges of generating explanations for various ensemble configurations. Our analysis uncovers nuanced complexity patterns influenced by various factors. For example, we demonstrate that under standard complexity assumptions like P$\neq$NP, interpreting ensembles remains intractable even when base models are of constant size. Surprisingly, the complexity changes drastically with the number of base models: small ensembles of decision trees are efficiently interpretable, whereas interpreting ensembles with even a constant number of linear models remains intractable. We believe that our findings provide a more robust foundation for understanding the interpretability of ensembles, emphasizing the benefits of examining it through a computational complexity lens.