TCP: a Benchmark for Temporal Constraint-Based Planning

TL;DR

The paper introduces TCP, a new benchmark that evaluates large language models' ability to perform complex temporal reasoning and planning within realistic dialogue scenarios, revealing current limitations.

Contribution

It presents a novel benchmark combining temporal reasoning and planning in dialogue form, with a comprehensive construction process and human validation.

Findings

State-of-the-art LLMs struggle with TCP tasks.

TCP reveals limitations in LLMs' temporal planning abilities.

Benchmark is open source for future research.

Abstract

Temporal reasoning and planning are essential capabilities for large language models (LLMs), yet most existing benchmarks evaluate them in isolation and under limited forms of complexity. To address this gap, we introduce the Temporal Constraint-based Planning (TCP) benchmark that jointly assesses both capabilities. Each instance in TCP features a naturalistic dialogue around a collaborative project, where diverse and interdependent temporal constraints are explicitly or implicitly expressed, and models must infer an optimal schedule that satisfies all constraints. To construct TCP, we generate abstract problem prototypes that are then paired with realistic scenarios from various domains and enriched into dialogues using an LLM. A human quality check is performed on a sampled subset to confirm the reliability of our benchmark. We evaluate state-of-the-art LLMs and find that even the strongest models may struggle with TCP, highlighting its difficulty and revealing limitations in LLMs' temporal constraint-based planning abilities. We analyze underlying failure cases, open source our benchmark, and hope our findings can inspire future research.