Grad-ELLM: Gradient-based Explanations for Decoder-only LLMs

TL;DR

Grad-ELLM introduces a gradient-based attribution method tailored for decoder-only transformer LLMs, improving faithfulness of input explanations without architectural changes.

Contribution

It proposes Grad-ELLM, a novel gradient-based attribution technique for decoder-only LLMs, along with new faithfulness metrics for better evaluation.

Findings

Grad-ELLM outperforms existing attribution methods in faithfulness.

The method works across multiple tasks and models.

New metrics enable fairer comparison of attribution methods.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet their black-box nature raises concerns about transparency and faithfulness. Input attribution methods aim to highlight each input token's contributions to the model's output, but existing approaches are typically model-agnostic, and do not focus on transformer-specific architectures, leading to limited faithfulness. To address this, we propose Grad-ELLM, a gradient-based attribution method for decoder-only transformer-based LLMs. By aggregating channel importance from gradients of the output logit with respect to attention layers and spatial importance from attention maps, Grad-ELLM generates heatmaps at each generation step without requiring architectural modifications. Additionally, we introduce two faithfulneses metrics $\pi$-Soft-NC and $\pi$-Soft-NS, which are modifications of Soft-NC/NS that provide fairer comparisons by controlling the amount of information kept when perturbing the text. We evaluate Grad-ELLM on sentiment classification, question answering, and open-generation tasks using different models. Experiment results show that Grad-ELLM consistently achieves superior faithfulness than other attribution methods.