BLAST: Benchmarking LLMs with ASP-based Structured Testing

TL;DR

BLAST introduces a new benchmarking framework and dataset to evaluate LLMs' ability to generate Answer Set Programming code, focusing on accuracy and semantic correctness.

Contribution

This paper presents the first dedicated ASP code generation benchmark with novel semantic metrics and empirical evaluation of multiple LLMs.

Findings

BLAST effectively measures LLM performance in ASP code generation.

Eight state-of-the-art LLMs were evaluated on graph-related ASP problems.

Results highlight strengths and limitations of current LLMs in declarative programming tasks.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs.