On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs

TL;DR

This paper reviews recent trends in NLP model interpretability, emphasizing stakeholder perspectives, analyzing research across fields, and highlighting disparities between developers and users to guide future interpretability methods.

Contribution

It provides a comprehensive analysis of interpretability paradigms, stakeholder needs, and research trends in NLP, informed by large-scale paper analysis using LLMs.

Findings

Significant disparities between NLP developers and non-developer users.

Explanations of internal model components are rarely used outside NLP.

Research trends vary across different scientific fields.

Abstract

Recent advancements in NLP systems, particularly with the introduction of LLMs, have led to widespread adoption of these systems by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and scientific research. This surge in usage has led to an explosion in NLP model interpretability and analysis research, accompanied by numerous technical surveys. Yet, these surveys often overlook the needs and perspectives of explanation stakeholders. In this paper, we address three fundamental questions: Why do we need interpretability, what are we interpreting, and how? By exploring these questions, we examine existing interpretability paradigms, their properties, and their relevance to different stakeholders. We further explore the practical implications of these paradigms by analyzing trends from the past decade across multiple research fields. To this end, we retrieved thousands of papers and employed an LLM to characterize them. Our analysis reveals significant disparities between NLP developers and non-developer users, as well as between research fields, underscoring the diverse needs of stakeholders. For example, explanations of internal model components are rarely used outside the NLP field. We hope this paper informs the future design, development, and application of methods that align with the objectives and requirements of various stakeholders.