Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?

TL;DR

This paper compares zero-shot LLM classifiers to LightGBM for loan default prediction, revealing that LLMs identify risk indicators but often produce explanations misaligned with actual feature importance, raising trust issues.

Contribution

It provides a systematic evaluation of LLMs versus classical models on structured financial data, highlighting limitations in their interpretability and reliability for risk assessment.

Findings

LLMs can identify key risk features but diverge in importance rankings.

Self-explanations from LLMs often do not match SHAP attributions.

Trustworthiness of LLM explanations in financial risk tasks is limited.

Abstract

Large Language Models (LLMs) are increasingly explored as flexible alternatives to classical machine learning models for classification tasks through zero-shot prompting. However, their suitability for structured tabular data remains underexplored, especially in high-stakes financial applications such as financial risk assessment. This study conducts a systematic comparison between zero-shot LLM-based classifiers and LightGBM, a state-of-the-art gradient-boosting model, on a real-world loan default prediction task. We evaluate their predictive performance, analyze feature attributions using SHAP, and assess the reliability of LLM-generated self-explanations. While LLMs are able to identify key financial risk indicators, their feature importance rankings diverge notably from LightGBM, and their self-explanations often fail to align with empirical SHAP attributions. These findings highlight the limitations of LLMs as standalone models for structured financial risk prediction and raise concerns about the trustworthiness of their self-generated explanations. Our results underscore the need for explainability audits, baseline comparisons with interpretable models, and human-in-the-loop oversight when deploying LLMs in risk-sensitive financial environments.