Can Large Language Models Reason and Optimize Under Constraints?

TL;DR

This paper evaluates large language models' ability to reason and optimize under constraints in power grid problems, revealing significant limitations and highlighting areas for improvement.

Contribution

It introduces a novel evaluation framework for testing LLMs on constrained reasoning tasks in power systems, exposing their current shortcomings.

Findings

State-of-the-art LLMs fail in most constrained reasoning tasks

Reasoning LLMs struggle with complex optimization scenarios

The work provides a rigorous testing environment for future LLM development

Abstract

Large Language Models (LLMs) have demonstrated great capabilities across diverse natural language tasks; yet their ability to solve abstraction and optimization problems with constraints remains scarcely explored. In this paper, we investigate whether LLMs can reason and optimize under the physical and operational constraints of Optimal Power Flow (OPF) problem. We introduce a challenging evaluation setup that requires a set of fundamental skills such as reasoning, structured input handling, arithmetic, and constrained optimization. Our evaluation reveals that SoTA LLMs fail in most of the tasks, and that reasoning LLMs still fail in the most complex settings. Our findings highlight critical gaps in LLMs' ability to handle structured reasoning under constraints, and this work provides a rigorous testing environment for developing more capable LLM assistants that can tackle real-world power grid optimization problems.