DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning

TL;DR

DeepRefine is a reinforcement learning-based system that iteratively improves large language model-compiled knowledge bases by diagnosing and refining their content, leading to better downstream task performance.

Contribution

It introduces a novel LLM reasoning model for knowledge refinement that uses abductive diagnosis and reinforcement learning to enhance knowledge base quality.

Findings

DeepRefine achieves consistent improvements over baseline methods.

The system effectively localizes and corrects knowledge defects.

Reinforcement learning with GBD reward optimizes refinement policies.

Abstract

Agent-compiled knowledge bases provide persistent external knowledge for large language model (LLM) agents in open-ended, knowledge-intensive downstream tasks. Yet their quality is systematically limited by \emph{incompleteness}, \emph{incorrectness}, and \emph{redundancy}, manifested as missing evidence or cross-document links, low-confidence or imprecise claims, and ambiguous or coreference resolution issues. Such defects compound under iterative use, degrading retrieval fidelity and downstream task performance. We present \textbf{DeepRefine}, a general LLM-based reasoning model for \emph{agent-compiled knowledge refinement} that improves the quality of any pre-constructed knowledge bases with user queries to make it more suitable for the downstream tasks. DeepRefine performs multi-turn interactions with the knowledge base and conducts abductive diagnosis over interaction history, localizes likely defects, and executes targeted refinement actions for incremental knowledge base updates. To optimize refinement policies of DeepRefine without gold references, we introduce a Gain-Beyond-Draft (GBD) reward and train the reasoning process end-to-end via reinforcement learning. Extensive experiments demonstrate consistent downstream gains over strong baselines.