Word Importance Explains How Prompts Affect Language Model Outputs

TL;DR

This paper introduces a method to explain how individual words in prompts influence large language model outputs by measuring their statistical impact, enhancing transparency and interpretability of LLMs.

Contribution

The study proposes a novel word importance measure based on permutation importance, applicable even without attention weights, to analyze prompt effects on LLM outputs.

Findings

Word importance scores correlate with suffix importance across models.

The method works with various scoring functions.

It improves understanding of prompt influence on LLM behavior.

Abstract

The emergence of large language models (LLMs) has revolutionized numerous applications across industries. However, their "black box" nature often hinders the understanding of how they make specific decisions, raising concerns about their transparency, reliability, and ethical use. This study presents a method to improve the explainability of LLMs by varying individual words in prompts to uncover their statistical impact on the model outputs. This approach, inspired by permutation importance for tabular data, masks each word in the system prompt and evaluates its effect on the outputs based on the available text scores aggregated over multiple user inputs. Unlike classical attention, word importance measures the impact of prompt words on arbitrarily-defined text scores, which enables decomposing the importance of words into the specific measures of interest--including bias, reading level, verbosity, etc. This procedure also enables measuring impact when attention weights are not available. To test the fidelity of this approach, we explore the effect of adding different suffixes to multiple different system prompts and comparing subsequent generations with different large language models. Results show that word importance scores are closely related to the expected suffix importances for multiple scoring functions.