A Counterfactual Diagnostic Framework for Explaining KS Deterioration in Credit Risk Model Validation

TL;DR

This paper introduces a counterfactual diagnostic framework to explain declines in KS statistics during credit risk model validation, improving interpretability and governance.

Contribution

It presents a standardized, sequential approach to diagnose KS deterioration sources, enhancing transparency over ad hoc methods.

Findings

Simulation shows improved interpretability of KS decline explanations.

Framework provides governance-relevant insights beyond threshold-based reviews.

Contributes to more consistent and transparent model validation processes.

Abstract

The Kolmogorov-Smirnov (KS) statistic is widely used in credit risk model monitoring and validation to assess discriminatory power. In practice, a material decline in KS often triggers governance review and requires validation teams to identify the breach source and the potential business risk. However, such diagnosis is frequently conducted on an ad hoc basis, relying on the judgment of individual validators rather than a standardized analytical framework. This paper proposes a counterfactual diagnostic framework for explaining KS deterioration in credit risk model validation. The framework sequentially attributes observed KS decline to sampling variability, portfolio composition change, covariate shift, and residual deterioration consistent with model drift, with explicit gateway conditions governing escalation at each stage. Simulation experiments demonstrate that the proposed approach provides more interpretable and governance-relevant explanations than threshold-based review alone, and contributes to more consistent, transparent, and defensible performance-breach assessment in credit risk model validation.