Enhancing Metacognitive AI: Knowledge-Graph Population with Graph-Theoretic LLM Enrichment

TL;DR

MetaKGEnrich is an automated pipeline that enhances LLMs with self-monitoring and knowledge repair capabilities using graph metrics, web evidence retrieval, and targeted question generation.

Contribution

Introduces MetaKGEnrich, a novel system integrating graph-theoretic metrics and web retrieval to enable self-directed knowledge repair in LLMs.

Findings

Improved answer quality in 80-87% of questions across datasets.

Effectively identifies sparse knowledge regions via graph metrics.

Demonstrates potential for AI to develop metacognitive learning abilities.

Abstract

Metacognition-the ability to monitor one's own knowledge state, spot gaps, and autonomously fill them--remains largely absent from modern AI. Here, we present MetaKGEnrich, a fully automated pipeline that endows large language model (LLM) applications with self-directed knowledge repair. The system (i) builds knowledge graphs from a seed query, (ii) detects sparse regions via seven graph metrics, (iii) has GPT-4o generate targeted questions, (iv) retrieves web evidence with Tavily and ingests it into Neo4j, and (v) re-answers the query with GraphRAG for GPT-4 to evaluate improvement. Tested on 30 queries from each of three widely-used datasets: Google Research Natural Questions, MS MARCO, and Hot-potQA. MetaKGEnrich improved answer quality in 80% of HotpotQA questions, 87% of Google Research Natural Questions and 83% of MS MARCO questions, while preserving well-supported regions. This proof of concept demonstrates how topological self-diagnosis plus targeted retrieval can advance AI toward humanlike metacognitive learning.