A Multi-Source Framework for Relational Validation of Large Language Models Using Expert-Curated Encyclopedic Sources

TL;DR

This paper presents a multi-source framework to evaluate the relational understanding of Large Language Models by comparing their knowledge graphs to expert-curated encyclopedias, revealing a significant relational deficit.

Contribution

The authors introduce a scalable, three-layer analytical framework for assessing LLMs' relational knowledge across diverse academic domains, highlighting a domain-dependent relational deficit.

Findings

LLMs recognize concepts but fail to reproduce their relational structure

Relational deficit varies significantly across domains

Complete relational failures occur in highly specialized fields

Abstract

This paper introduces a novel, multi-source framework for the relational validation of Large Language Models (LLMs). While existing benchmarks have demonstrated LLMs' proficiency at factual recall, their ability to understand and reproduce the intricate web of relationships that defines a domain's conceptual structure remains largely unexplored. Our three-layer analytical framework provides a scalable and robust methodology for assessing the depth of an LLM's knowledge across diverse academic domains. By comparing LLM-generated knowledge graphs to expert-curated encyclopedias, we reveal a consistent and significant ``relational deficit'': LLMs recognize domain-specific concepts but consistently fail to reproduce their relational structure. Our findings highlight the need for more sophisticated evaluation metrics that go beyond simple accuracy and assess the relational integrity of an LLM's knowledge. We demonstrate that this deficit is highly domain-dependent, with performance varying significantly across ten specialized encyclopedias spanning sociology, political science, philosophy, and other fields. The cases of complete relational failure in the most specialized domains are particularly revealing, suggesting that the LLM's internal knowledge representation is not aligned with the conceptual structures of these fields. This has significant implications for the deployment of LLMs in high-stakes applications that require a deep, nuanced understanding of domain-specific knowledge.