CP-Agent: Agentic Constraint Programming

TL;DR

CP-Agent is a Python-based agentic system that iteratively refines constraint models from natural language prompts, achieving perfect accuracy on a benchmark of 101 problems, demonstrating the effectiveness of minimal guidance.

Contribution

This paper introduces CP-Agent, a novel agentic framework for translating natural language into formal constraint models using iterative execution and refinement.

Findings

Achieved perfect accuracy on all 101 benchmark problems after clarification.

Minimal guidance outperforms detailed procedural scaffolding.

Explicit task management tools can have mixed effects on modeling tasks.

Abstract

The translation of natural language to formal constraint models requires expertise in the problem domain and modeling frameworks. To explore the effectiveness of agentic workflows, we propose CP-Agent, a Python coding agent that uses the ReAct framework with a persistent IPython kernel. We provide the relevant domain knowledge as a project prompt of under 50 lines. The algorithm works by iteratively executing code, observing the solver's feedback, and refining constraint models based on execution results. We evaluate CP-Agent on 101 constraint programming problems from CP-Bench. We made minor changes to the benchmark to address systematic ambiguities in the problem specifications and errors in the ground-truth models. On the clarified benchmark, CP-Agent achieves perfect accuracy on all 101 problems. Our experiments show that minimal guidance outperforms detailed procedural scaffolding. Our experiments also show that explicit task management tools can have both positive and negative effects on focused modeling tasks.