KoRe: Compact Knowledge Representations for Large Language Models

TL;DR

KoRe introduces a method to encode 1-hop sub-graphs into compact tokens, enabling efficient integration of knowledge graphs into large language models with reduced token usage and maintained performance.

Contribution

The paper presents KoRe, a novel approach for embedding knowledge graph sub-graphs into LLMs using discrete tokens, reducing token count and improving knowledge grounding.

Findings

Achieved up to 10x reduction in token usage.

Demonstrated competitive performance on three benchmarks.

Showed effective grounding of LLMs with compact KG representations.

Abstract

Modern Large Language Models (LLMs) have shown impressive performances in user-facing tasks such as question answering, as well as consistent improvements in reasoning capabilities. Still, the way these models encode knowledge seems inherently flawed: by design, LLMs encode world-knowledge within their parameters. This way of representing knowledge is inherently opaque, difficult to debug and update, and prone to hallucinations. On the other hand, Knowledge Graphs can provide human-readable and easily editable world knowledge representations, and their application in knowledge-intensive tasks has consistently proven beneficial to downstream performance. Nonetheless, current integration techniques require extensive retraining or finetuning. To overcome this issue, we introduce KoRe, a methodology to encode 1-hop sub-graphs into compact discrete knowledge tokens and inject them into a LLM backbone. We test the proposed approach on three established benchmarks, and report competitive performances coupled with a significant reduction (up to 10x) in token usage. Our results show that compact discrete KG representations can efficiently and effectively be used to ground modern LLMs.