KLoB: a Benchmark for Assessing Knowledge Locating Methods in Language Models

TL;DR

KLoB is a new benchmark designed to evaluate the effectiveness and reliability of knowledge locating methods in language models, addressing key properties and testing the locality hypothesis of factual knowledge.

Contribution

The paper introduces KLoB, a comprehensive benchmark for assessing knowledge locating methods and for testing the locality hypothesis in language models.

Findings

KLoB effectively evaluates existing locating methods.

It provides insights into the validity of the locality hypothesis.

KLoB is publicly available for research use.

Abstract

Recently, Locate-Then-Edit paradigm has emerged as one of the main approaches in changing factual knowledge stored in the Language models. However, there is a lack of research on whether present locating methods can pinpoint the exact parameters embedding the desired knowledge. Moreover, although many researchers have questioned the validity of locality hypothesis of factual knowledge, no method is provided to test the a hypothesis for more in-depth discussion and research. Therefore, we introduce KLoB, a benchmark examining three essential properties that a reliable knowledge locating method should satisfy. KLoB can serve as a benchmark for evaluating existing locating methods in language models, and can contributes a method to reassessing the validity of locality hypothesis of factual knowledge. KLoB is publicly available at an anonymous GitHub: \url{https://github.com/anon6662/KLoB}.