Test-driven Software Experimentation with LASSO: an LLM Prompt Benchmarking Example

TL;DR

LASSO is a versatile platform that facilitates rapid, standardized test-driven experiments in software engineering, enabling detailed analysis of runtime behavior and properties, exemplified through benchmarking LLMs for code generation.

Contribution

This paper introduces LASSO, a novel platform with domain-specific languages for conducting standardized, executable test-driven software experiments efficiently.

Findings

LASSO enables efficient evaluation of runtime semantics.

The platform supports reusable and extensible experiment scripts.

Demonstrated with an LLM code generation reliability study.

Abstract

Empirical software engineering faces a critical gap: the lack of standardized tools for rapid development and execution of Test-Driven Software Experiments (TDSEs) -- that is, experiments that involve the execution of software subjects and the observation and analysis of their "de facto" run-time behavior. In this paper we present a general-purpose analysis platform called LASSO that provides a minimal set of domain-specific languages and data structures to conduct TDSEs. By empowering users with an executable scripting language to design and execute TDSEs, LASSO enables efficient evaluation of run-time semantics and execution characteristics in addition to statically determined properties. We present an example TDSE that demonstrates the practical benefits of LASSO's scripting capabilities for assessing the reliability of LLMs for code generation by means of a self-contained, reusable and extensible study script. The LASSO platform and live pipeline examples are publicly available at: https://softwareobservatorium.github.io/.