A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs

TL;DR

This paper introduces a benchmark to detect both factual and metalinguistic disagreements between large language models and knowledge graphs, highlighting the importance of understanding language meaning differences in NLP evaluations.

Contribution

It proposes a novel benchmark for identifying metalinguistic disagreements, addressing a gap in current evaluation methods for LLMs and knowledge graphs.

Findings

Metalinguistic disagreements occur between LLMs and KGs

The benchmark can differentiate factual from metalinguistic errors

Initial proof of concept is available on Github

Abstract

Evaluating large language models (LLMs) for tasks like fact extraction in support of knowledge graph construction frequently involves computing accuracy metrics using a ground truth benchmark based on a knowledge graph (KG). These evaluations assume that errors represent factual disagreements. However, human discourse frequently features metalinguistic disagreement, where agents differ not on facts but on the meaning of the language used to express them. Given the complexity of natural language processing and generation using LLMs, we ask: do metalinguistic disagreements occur between LLMs and KGs? Based on an investigation using the T-REx knowledge alignment dataset, we hypothesize that metalinguistic disagreement does in fact occur between LLMs and KGs, with potential relevance for the practice of knowledge graph engineering. We propose a benchmark for evaluating the detection of factual and metalinguistic disagreements between LLMs and KGs. An initial proof of concept of such a benchmark is available on Github.