PDDL-Mind: Large Language Models are Capable on Belief Reasoning with Reliable State Tracking

TL;DR

PDDL-Mind enhances belief reasoning in large language models by explicitly tracking environment states with a neuro-symbolic approach, significantly improving performance on theory-of-mind benchmarks.

Contribution

It introduces PDDL-Mind, a neuro-symbolic framework that explicitly models environment states to improve LLMs' belief reasoning capabilities.

Findings

PDDL-Mind achieves over 5% accuracy improvement on ToM benchmarks.

Explicit state tracking reduces LLMs' belief reasoning errors.

The framework outperforms existing methods on multiple ToM datasets.

Abstract

Large language models (LLMs) perform substantially below human level on existing theory-of-mind (ToM) benchmarks, even when augmented with chain-of-thought prompting or probabilistic belief updates. We argue that these failures primarily arise from unreliable implicit state tracking rather than limitations in high-level reasoning. We introduce PDDL-Mind, a neuro-symbolic framework that decouples environment state evolution from belief inference. By translating narrative descriptions into explicit states and actions expressed in Planning Domain Definition Language (PDDL), and by verifying action-induced state transitions against a predefined domain, PDDL-Mind provides LLMs with a logically consistent and explicit representation of world states for ToM tasks. Experiments on MMToM-QA, MuMA and FanToM show that PDDL-Mind achieves over 5% absolute accuracy gain over the best existing state-of-the-art method on ToM benchmark questions.