Model-First Reasoning LLM Agents: Reducing Hallucinations through Explicit Problem Modeling

TL;DR

This paper introduces Model-First Reasoning (MFR), a two-phase approach for LLMs that explicitly models problems before planning, significantly reducing hallucinations and improving solution quality across various domains.

Contribution

The paper proposes MFR, a novel explicit problem modeling paradigm for LLMs, which enhances planning accuracy and interpretability over existing implicit methods.

Findings

MFR reduces constraint violations across multiple domains.

Explicit modeling is critical for improved LLM planning performance.

MFR outperforms Chain-of-Thought and ReAct in solution quality.

Abstract

Large Language Models (LLMs) often struggle with complex multi-step planning tasks, showing high rates of constraint violations and inconsistent solutions. Existing strategies such as Chain-of-Thought and ReAct rely on implicit state tracking and lack an explicit problem representation. Inspired by classical AI planning, we propose Model-First Reasoning (MFR), a two-phase paradigm in which the LLM first constructs an explicit model of the problem, defining entities, state variables, actions, and constraints, before generating a solution plan. Across multiple planning domains, including medical scheduling, route planning, resource allocation, logic puzzles, and procedural synthesis, MFR reduces constraint violations and improves solution quality compared to Chain-of-Thought and ReAct. Ablation studies show that the explicit modeling phase is critical for these gains. Our results suggest that many LLM planning failures stem from representational deficiencies rather than reasoning limitations, highlighting explicit modeling as a key component for robust and interpretable AI agents. All prompts, evaluation procedures, and task datasets are documented to facilitate reproducibility.