ASP-Bench: From Natural Language to Logic Programs

TL;DR

ASP-Bench is a comprehensive benchmark for translating natural language specifications into Answer Set Programs, covering diverse reasoning features and enabling evaluation of different systems' modeling capabilities.

Contribution

Introduces ASP-Bench, a new benchmark with 128 natural language problems and systematic coverage of ASP features for evaluating translation systems.

Findings

ReAct framework achieves full saturation on the benchmark.

Feedback-driven iterative refinement improves natural language to ASP translation.

Analysis reveals key factors influencing problem modeling difficulty.

Abstract

Automating the translation of natural-language specifications into logic programs is a challenging task that affects neurosymbolic engineering. We present ASP-Bench, a benchmark comprising 128 natural language problem instances, 64 base problems with easy and hard variants. It evaluates systems that translate natural-language problems into Answer Set Programs (ASPs), a prominent form of logic programming. It provides systematic coverage of ASP features, including choice rules, aggregates, and optimization. Each problem includes reference validators that check whether solutions satisfy the problem specification. We characterize problems along seven largely independent reasoning aspects (optimization, temporal reasoning, default logic, resource allocation, recursion, spatial reasoning, and quantitative complexity), providing a multidimensional view of modeling difficulty. We test the benchmark using an agentic approach based on the ReAct (Reason and Act) framework, which achieves full saturation, demonstrating that feedback-driven iterative refinement with solver feedback provides a reliable and robust approach for modeling natural language in ASP. Our analysis across multiple agent runs enables us to gain insights into what determines a problem's modeling hardness.