Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods

TL;DR

This paper introduces a unified framework to evaluate and compare attribution methods for understanding the parametric knowledge stored in language models, highlighting the complementary strengths of Instance and Neuron Attribution techniques.

Contribution

The study develops a novel evaluation framework and new attribution methods, providing systematic comparison and insights into the knowledge revealed by IA and NA in language models.

Findings

NA reveals more diverse and comprehensive knowledge

IA offers unique insights not captured by NA

Combining IA and NA can enhance understanding of LM knowledge

Abstract

Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. The increasing scalability of LMs, however, poses significant challenges for understanding a model's inner workings and further for updating or correcting this embedded knowledge without the significant cost of retraining. This underscores the importance of unveiling exactly what knowledge is stored and its association with specific model components. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge, though they have not been compared systematically. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA. To align the results of the methods we introduce the attribution method NA-Instances to apply NA for retrieving influential training instances, and IA-Neurons to discover important neurons of influential instances discovered by IA. We further propose a comprehensive list of faithfulness tests to evaluate the comprehensiveness and sufficiency of the explanations provided by both methods. Through extensive experiments and analysis, we demonstrate that NA generally reveals more diverse and comprehensive information regarding the LM's parametric knowledge compared to IA. Nevertheless, IA provides unique and valuable insights into the LM's parametric knowledge, which are not revealed by NA. Our findings further suggest the potential of a synergistic approach of combining the diverse findings of IA and NA for a more holistic understanding of an LM's parametric knowledge.