Distilling a Neural Network Into a Soft Decision Tree

TL;DR

This paper presents a method to convert trained neural networks into soft decision trees, making model decisions more interpretable while maintaining or improving generalization performance.

Contribution

It introduces a novel approach to distill neural network knowledge into soft decision trees, enhancing interpretability without sacrificing accuracy.

Findings

Soft decision trees generalize better than those learned directly from data.

The method improves interpretability of neural network decisions.

Distilled trees retain high classification accuracy.

Abstract

Deep neural networks have proved to be a very effective way to perform classification tasks. They excel when the input data is high dimensional, the relationship between the input and the output is complicated, and the number of labeled training examples is large. But it is hard to explain why a learned network makes a particular classification decision on a particular test case. This is due to their reliance on distributed hierarchical representations. If we could take the knowledge acquired by the neural net and express the same knowledge in a model that relies on hierarchical decisions instead, explaining a particular decision would be much easier. We describe a way of using a trained neural net to create a type of soft decision tree that generalizes better than one learned directly from the training data.