Knowledge Graph Representations for LLM-Based Policy Compliance Reasoning
Wilder Baldwin, Sepideh Ghanavati

TL;DR
This paper introduces a framework that constructs knowledge graphs from AI policy documents to enhance large language models' ability to answer policy-related questions, improving reasoning accuracy.
Contribution
It presents a novel method for building knowledge graphs from AI policies and demonstrates their effectiveness in improving LLM-based policy reasoning tasks.
Findings
KG augmentation improves LLM performance across tasks
Open LLM-discovered schema matches or exceeds formal ontology
Evaluation on 42 policy QA tasks shows significant gains
Abstract
The risks posed by AI features are increasing as they are rapidly integrated into software applications. In response, regulations and standards for safe and secure AI have been proposed. In this paper, we present an agentic framework that constructs knowledge graphs (KGs) from AI policy documents and retrieves policy-relevant information to answer questions. We build KGs from three AI risk-related polices under two ontology schemas, and then evaluate five LLMs on 42 policy QA tasks spanning six reasoning types, from entity lookup to cross-policy inference, using both heuristic scoring and an LLM-as-judge. KG augmentation improves scores for all five models, and an open, LLM-discovered schema matches or exceeds the formal ontology.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
