Could Large Language Models work as Post-hoc Explainability Tools in Credit Risk Models?

TL;DR

This study assesses whether large language models can effectively serve as post-hoc explainability tools for credit risk models, focusing on their ability to replicate feature importance and generate human-readable explanations.

Contribution

It provides an empirical evaluation of LLMs as narrative interfaces for credit risk explanations, comparing their performance to traditional attribution methods.

Findings

LLMs reliably reproduce feature importance rankings with controlled prompts.

LLMs show limited alignment when generating autonomous explanations.

LLMs are better suited as narrative interfaces than as substitutes for formal attribution methods.

Abstract

Large language models (LLMs) have shown promise in translating model-based explanations into human-readable narratives. This study evaluates whether LLMs can serve as post-hoc explainability interfaces for credit risk models, focusing on their ability to preserve feature-importance rankings and generate autonomous explanations. Using a LendingClub dataset, we compare LLM outputs with SHAP and coefficient-based attributions on three major LLMs, including GPT-4-turbo, Claude-Sonnet-4.5, and Gemini-2.5-Flash. Results indicate that LLMs reliably reproduce reference rankings under controlled prompts but show limited alignment when generating explanations autonomously. These findings suggest that LLMs are best deployed as narrative interfaces rather than substitutes for formal attribution methods in credit risk governance.