FLAME: Empowering Frozen LLMs for Knowledge Graph Completion

TL;DR

FLAME leverages frozen LLMs to enhance knowledge graph completion efficiently by extracting context-aware features, achieving significant improvements over non-fine-tuned models and comparable performance to fine-tuning with much lower resource costs.

Contribution

The paper introduces FLAME, a novel framework that extracts hidden states from frozen LLMs for knowledge graph completion, combining efficiency with high performance.

Findings

47% improvement over non-fine-tuned LLM baselines

First to match fine-tuned performance with 188x memory savings

26.11x speedup in training and inference

Abstract

Traditional knowledge graph completion (KGC) methods rely solely on structural information and struggle with sparsity, while Large Language Models (LLMs) address these limitations through rich world knowledge and strong context modeling. Fine-tuning LLMs is effective but costly, while non-fine-tuned LLMs are efficient but suboptimal. To address this trade-off, we propose \textbf{FLAME}, a framework that extracts context-aware hidden states from intermediate layers of frozen LLMs to train data-efficient KGC classifiers. We bridge LLM-KG semantic gaps via subgraph-based entity descriptions and employ sliced mutual information (SMI) to quantify task-relevant information in representations. Experiments demonstrate that FLAME achieves 47\% improvement over non-fine-tuned LLM baselines and, to our knowledge, is the first to achieve fine-tuned performance with $188\times$ memory efficiency and $26.11\times$ speedup.