LLASP: Fine-tuning Large Language Models for Answer Set Programming

TL;DR

This paper introduces LLASP, a fine-tuned model designed to improve the generation of Answer Set Programming code by leveraging a specialized dataset and fine-tuning techniques, significantly enhancing semantic correctness.

Contribution

The paper presents a novel fine-tuning approach with a dedicated ASP dataset, resulting in superior ASP code generation compared to general LLMs.

Findings

LLASP outperforms non-fine-tuned models in ASP code quality

Fine-tuning improves semantic accuracy of generated ASP programs

General LLMs show inadequate performance in ASP code generation

Abstract

Recently, Large Language Models (LLMs) have showcased their potential in various natural language processing tasks, including code generation. However, while significant progress has been made in adapting LLMs to generate code for several imperative programming languages and tasks, there remains a notable gap in their application to declarative formalisms, such as Answer Set Programming (ASP). In this paper, we move a step towards exploring the capabilities of LLMs for ASP code generation. First, we perform a systematic evaluation of several state-of-the-art LLMs. Despite their power in terms of number of parameters, training data and computational resources, empirical results demonstrate inadequate performances in generating correct ASP programs. Therefore, we propose LLASP, a fine-tuned lightweight model specifically trained to encode fundamental ASP program patterns. To this aim, we create an ad-hoc dataset covering a wide variety of fundamental problem specifications that can be encoded in ASP. Our experiments demonstrate that the quality of ASP programs generated by LLASP is remarkable. This holds true not only when compared to the non-fine-tuned counterpart but also when compared to the majority of eager LLM candidates, particularly from a semantic perspective. All the code and data used to perform the experiments are publicly available at https://anonymous.4open.science/r/LLASP-D86C/.