Auto-Formulating Dynamic Programming Problems with Large Language Models

TL;DR

This paper introduces DPLM, a specialized large language model for automating dynamic programming problem formulation, and presents DP-Bench, a benchmark for evaluating such models on textbook DP problems.

Contribution

The paper develops DPLM, a 7B-parameter LLM tailored for DP problems, and proposes DualReflect, a synthetic data generation pipeline that balances diversity and correctness.

Findings

DPLM achieves performance comparable to state-of-the-art LLMs on DP tasks.

DualReflect effectively scales training data using forward and backward generation methods.

Backward generation is more reliable in low-data regimes, while forward generation enhances diversity at scale.

Abstract

Dynamic programming (DP) is a fundamental method in operations research, but formulating DP models has traditionally required expert knowledge of both the problem context and DP techniques. Large Language Models (LLMs) offer the potential to automate this process. However, DP problems pose unique challenges due to their inherently stochastic transitions and the limited availability of training data. These factors make it difficult to directly apply existing LLM-based models or frameworks developed for other optimization problems, such as linear or integer programming. We introduce DP-Bench, the first benchmark covering a wide range of textbook-level DP problems to enable systematic evaluation. We present Dynamic Programming Language Model (DPLM), a 7B-parameter specialized model that achieves performance comparable to state-of-the-art LLMs like OpenAI's o1 and DeepSeek-R1, and surpasses them on hard problems. Central to DPLM's effectiveness is DualReflect, our novel synthetic data generation pipeline, designed to scale up training data from a limited set of initial examples. DualReflect combines forward generation for diversity and backward generation for reliability. Our results reveal a key insight: backward generation is favored in low-data regimes for its strong correctness guarantees, while forward generation, though lacking such guarantees, becomes increasingly valuable at scale for introducing diverse formulations. This trade-off highlights the complementary strengths of both approaches and the importance of combining them.