Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention

TL;DR

This paper introduces SparseCBM, a sparsity-guided framework that offers a comprehensive, multi-layered interpretation of large language models and enables dynamic inference-time interventions for improved transparency and accuracy.

Contribution

The paper presents a novel sparsity-based approach that unifies input, subnetwork, and concept explanations, along with a new inference-time intervention dimension for LLM interpretability.

Findings

Provides a holistic interpretation of LLMs across multiple layers.

Enables dynamic adjustments during inference to improve model performance.

Demonstrates effectiveness through empirical evaluations on real datasets.

Abstract

Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. However, the enigmatic ``black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. While past approaches, such as attention visualization, pivotal subnetwork extraction, and concept-based analyses, offer some insight, they often focus on either local or global explanations within a single dimension, occasionally falling short in providing comprehensive clarity. In response, we propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs. Our framework, termed SparseCBM, innovatively integrates sparsity to elucidate three intertwined layers of interpretation: input, subnetwork, and concept levels. In addition, the newly introduced dimension of interpretable inference-time intervention facilitates dynamic adjustments to the model during deployment. Through rigorous empirical evaluations on real-world datasets, we demonstrate that SparseCBM delivers a profound understanding of LLM behaviors, setting it apart in both interpreting and ameliorating model inaccuracies. Codes are provided in supplements.