From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality

TL;DR

This study systematically investigates how different factors like model choice, XAI methods, LLM selection, and prompting strategies influence the quality of natural language explanations generated for time-series forecasting, revealing key insights into their effectiveness.

Contribution

It provides a comprehensive factorial analysis of explanation quality factors in LLM-generated explanations, highlighting the dominance of LLM choice and revealing the interpretability paradox in model performance.

Findings

XAI offers limited benefits over no-XAI for non-expert users.

LLM choice significantly impacts explanation quality, with DeepSeek-R1 outperforming others.

SARIMAX models produce lower explanation quality despite higher accuracy.

Abstract

Explainable AI (XAI) methods like SHAP and LIME produce numerical feature attributions that remain inaccessible to non expert users. Prior work has shown that Large Language Models (LLMs) can transform these outputs into natural language explanations (NLEs), but it remains unclear which factors contribute to high-quality explanations. We present a systematic factorial study investigating how Forecasting model choice, XAI method, LLM selection, and prompting strategy affect NLE quality. Our design spans four models (XGBoost (XGB), Random Forest (RF), Multilayer Perceptron (MLP), and SARIMAX - comparing black-box Machine-Learning (ML) against classical time-series approaches), three XAI conditions (SHAP, LIME, and a no-XAI baseline), three LLMs (GPT-4o, Llama-3-8B, DeepSeek-R1), and eight prompting strategies. Using G-Eval, an LLM-as-a-judge evaluation method, with dual LLM judges and four evaluation criteria, we evaluate 660 explanations for time-series forecasting. Our results suggest that: (1) XAI provides only small improvements over no-XAI baselines, and only for expert audiences; (2) LLM choice dominates all other factors, with DeepSeek-R1 outperforming GPT-4o and Llama-3; (3) we observe an interpretability paradox: in our setting, SARIMAX yielded lower NLE quality than ML models despite higher prediction accuracy; (4) zero-shot prompting is competitive with self-consistency at 7-times lower cost; and (5) chain-of-thought hurts rather than helps.