Knowledge Distillation Decision Tree for Unravelling Black-box Machine Learning Models

TL;DR

This paper introduces a novel method called KDDT that distills knowledge from black-box models into decision trees, enhancing interpretability while maintaining accuracy and stability, supported by theoretical foundations and empirical validation.

Contribution

The paper proposes the KDDT method for interpreting black-box models through stable and predictive decision trees, including a hybrid version with an efficient construction algorithm.

Findings

KDDT achieves structural stability under mild assumptions.

Hybrid KDDT balances simplicity and predictivity effectively.

Simulation and real-data studies confirm the method's accuracy and reliability.

Abstract

Machine learning models, particularly the black-box models, are widely favored for their outstanding predictive capabilities. However, they often face scrutiny and criticism due to the lack of interpretability. Paradoxically, their strong predictive capabilities may indicate a deep understanding of the underlying data, implying significant potential for interpretation. Leveraging the emerging concept of knowledge distillation, we introduce the method of knowledge distillation decision tree (KDDT). This method enables the distillation of knowledge about the data from a black-box model into a decision tree, thereby facilitating the interpretation of the black-box model. Essential attributes for a good interpretable model include simplicity, stability, and predictivity. The primary challenge of constructing interpretable tree lies in ensuring structural stability under the randomness of the training data. KDDT is developed with the theoretical foundations demonstrating that structure stability can be achieved under mild assumptions. Furthermore, we propose the hybrid KDDT to achieve both simplicity and predictivity. An efficient algorithm is provided for constructing the hybrid KDDT. Simulation studies and a real-data analysis validate the hybrid KDDT's capability to deliver accurate and reliable interpretations. KDDT is an excellent interpretable model with great potential for practical applications.