Beyond Individual Facts: Investigating Categorical Knowledge Locality of Taxonomy and Meronomy Concepts in GPT Models

TL;DR

This paper explores how concepts and categories, rather than individual facts, are represented within GPT models, revealing that related categories tend to share similar importance regions but lack fine-grained localization.

Contribution

It introduces DARC, a novel dataset of hierarchical concepts, and applies causal mediation analysis to investigate the regional importance of categories in GPT models.

Findings

Related categories share similar importance regions

Fine-grained localization of individual categories is not evident

The study broadens understanding of knowledge representation in GPTs

Abstract

The location of knowledge within Generative Pre-trained Transformer (GPT)-like models has seen extensive recent investigation. However, much of the work is focused towards determining locations of individual facts, with the end goal being the editing of facts that are outdated, erroneous, or otherwise harmful, without the time and expense of retraining the entire model. In this work, we investigate a broader view of knowledge location, that of concepts or clusters of related information, instead of disparate individual facts. To do this, we first curate a novel dataset, called DARC, that includes a total of 34 concepts of ~120K factual statements divided into two types of hierarchical categories, namely taxonomy and meronomy. Next, we utilize existing causal mediation analysis methods developed for determining regions of importance for individual facts and apply them to a series of related categories to provide detailed investigation into whether concepts are associated with distinct regions within these models. We find that related categories exhibit similar areas of importance in contrast to less similar categories. However, fine-grained localization of individual category subsets to specific regions is not apparent.