Using Knowledge Distillation to improve interpretable models in a retail banking context

TL;DR

This paper reviews knowledge distillation techniques and demonstrates their potential to enhance simple, interpretable models in retail banking without increasing complexity, using real-world data and use cases.

Contribution

It highlights three main approaches to knowledge distillation and tests their effectiveness in retail banking applications, showing performance improvements.

Findings

Knowledge distillation improves model accuracy in banking tasks.

Soft Targets, Sample Selection, Data Augmentation are effective techniques.

Enhanced models maintain interpretability and simplicity.

Abstract

This article sets forth a review of knowledge distillation techniques with a focus on their applicability to retail banking contexts. Predictive machine learning algorithms used in banking environments, especially in risk and control functions, are generally subject to regulatory and technical constraints limiting their complexity. Knowledge distillation gives the opportunity to improve the performances of simple models without burdening their application, using the results of other - generally more complex and better-performing - models. Parsing recent advances in this field, we highlight three main approaches: Soft Targets, Sample Selection and Data Augmentation. We assess the relevance of a subset of such techniques by applying them to open source datasets, before putting them to the test on the use cases of BPCE, a major French institution in the retail banking sector. As such, we demonstrate the potential of knowledge distillation to improve the performance of these models without altering their form and simplicity.