Interpretable Representation Learning for Additive Rule Ensembles

TL;DR

This paper introduces a novel method for learning interpretable rule ensembles using logical propositions with learnable sparse linear transformations, enabling more flexible decision regions and reducing model complexity.

Contribution

It extends classical rule ensembles by incorporating learnable sparse linear transformations, allowing for more general polytopic decision regions with improved interpretability and efficiency.

Findings

Achieves similar test risk to state-of-the-art methods.

Reduces model complexity significantly.

Efficiently constructs rule ensembles across multiple datasets.

Abstract

Small additive ensembles of symbolic rules offer interpretable prediction models. Traditionally, these ensembles use rule conditions based on conjunctions of simple threshold propositions $x \geq t$ on a single input variable $x$ and threshold $t$, resulting geometrically in axis-parallel polytopes as decision regions. While this form ensures a high degree of interpretability for individual rules and can be learned efficiently using the gradient boosting approach, it relies on having access to a curated set of expressive and ideally independent input features so that a small ensemble of axis-parallel regions can describe the target variable well. Absent such features, reaching sufficient accuracy requires increasing the number and complexity of individual rules, which diminishes the interpretability of the model. Here, we extend classical rule ensembles by introducing logical propositions with learnable sparse linear transformations of input variables, i.e., propositions of the form $\mathbf{x}^\mathrm{T}\mathbf{w} \geq t$, where $\mathbf{w}$ is a learnable sparse weight vector, enabling decision regions as general polytopes with oblique faces. We propose a learning method using sequential greedy optimization based on an iteratively reweighted formulation of logistic regression. Experimental results demonstrate that the proposed method efficiently constructs rule ensembles with the same test risk as state-of-the-art methods while significantly reducing model complexity across ten benchmark datasets.