Model Interpretation and Explainability: Towards Creating Transparency in Prediction Models

TL;DR

This paper investigates the challenges of model explainability in prediction models, analyzing different methods and scenarios to identify inconsistencies in feature importance and suggesting directions for future research.

Contribution

It provides a comparative analysis of prediction methods and explainability techniques, highlighting inconsistencies between static and dynamic feature importance assessments.

Findings

Inconsistencies found between static and dynamic feature importance.

Comparison of four prediction methods on credit dataset.

Highlights need for improved explainability techniques.

Abstract

Explainable AI (XAI) has a counterpart in analytical modeling which we refer to as model explainability. We tackle the issue of model explainability in the context of prediction models. We analyze a dataset of loans from a credit card company and apply three stages: execute and compare four different prediction methods, apply the best known explainability techniques in the current literature to the model training sets to identify feature importance (FI) (static case), and finally to cross-check whether the FI set holds up under what if prediction scenarios for continuous and categorical variables (dynamic case). We found inconsistency in FI identification between the static and dynamic cases. We summarize the state of the art in model explainability and suggest further research to advance the field.