Overcoming Knowledge Discrepancies: Structuring Reasoning Threads through Knowledge Balancing in Interactive Scenarios

TL;DR

This paper introduces ReT-Eval, a two-phase framework that constructs and refines reasoning threads by balancing structured domain knowledge and large language model insights, improving interactive problem solving.

Contribution

It presents a novel two-phase reasoning framework that extracts, enriches, and prunes knowledge-based reasoning threads to enhance user understanding and model effectiveness.

Findings

ReT-Eval outperforms existing reasoning models in experiments.

The framework improves user understanding in interactive scenarios.

Knowledge balancing enhances reasoning coherence and effectiveness.

Abstract

Reasoning in interactive problem solving scenarios requires models to construct reasoning threads that reflect user understanding and align with structured domain knowledge. However, current reasoning models often lack explicit semantic hierarchies, user-domain knowledge alignment, and principled mechanisms to prune reasoning threads for effectiveness. These limitations result in lengthy generic output that does not guide users through goal-oriented reasoning steps. To address this, we propose a prototype-inspired, two-phases Reasoning-Threads-Evaluation (ReT-Eval) framework, drawing inspiration from human-like reasoning strategies that emphasize structured knowledge reuse. In the first phase, semantically relevant knowledge structures are extracted from a sparse domain knowledge graph using a graph neural network and enriched with intrinsic large language model knowledge to resolve knowledge discrepancies. In the second phase, these threads are evaluated and pruned using a reward-guided strategy aimed at maintaining semantic coherence to generate effective reasoning threads. Experiments and expert evaluations show that ReT-Eval enhances user understanding and outperforms state-of-the-art reasoning models.