Distilling Knowledge from Deep Networks with Applications to Healthcare Domain

TL;DR

This paper presents a novel knowledge-distillation method called Interpretable Mimic Learning that enables deep learning models in healthcare to produce interpretable features for clinical decision-making without sacrificing predictive performance.

Contribution

The paper introduces a new framework that distills deep learning models into interpretable models using Gradient Boosting Trees, improving interpretability in healthcare applications.

Findings

Achieves comparable or better predictive performance than original deep models.

Provides meaningful, interpretable phenotype features for clinicians.

Demonstrates effectiveness on real-world clinical time-series data.

Abstract

Exponential growth in Electronic Healthcare Records (EHR) has resulted in new opportunities and urgent needs for discovery of meaningful data-driven representations and patterns of diseases in Computational Phenotyping research. Deep Learning models have shown superior performance for robust prediction in computational phenotyping tasks, but suffer from the issue of model interpretability which is crucial for clinicians involved in decision-making. In this paper, we introduce a novel knowledge-distillation approach called Interpretable Mimic Learning, to learn interpretable phenotype features for making robust prediction while mimicking the performance of deep learning models. Our framework uses Gradient Boosting Trees to learn interpretable features from deep learning models such as Stacked Denoising Autoencoder and Long Short-Term Memory. Exhaustive experiments on a real-world clinical time-series dataset show that our method obtains similar or better performance than the deep learning models, and it provides interpretable phenotypes for clinical decision making.