N-Version Assessment and Enhancement of Generative AI

TL;DR

This paper introduces a differential GAI approach that generates multiple code versions for comparative analysis, enhancing the reliability of GAI outputs and proposing a platform for large-scale evaluation.

Contribution

It presents the D-GAI method and the LASSO platform to improve verification of GAI-generated code through version diversity and large-scale analysis.

Findings

D-GAI improves reliability of GAI outputs.

LASSO enables large-scale evaluation of code versions.

Differential analysis enhances trust in GAI-generated artifacts.

Abstract

Generative AI (GAI) holds great potential to improve software engineering productivity, but its untrustworthy outputs, particularly in code synthesis, pose significant challenges. The need for extensive verification and validation (V&V) of GAI-generated artifacts may undermine the potential productivity gains. This paper proposes a way of mitigating these risks by exploiting GAI's ability to generate multiple versions of code and tests to facilitate comparative analysis across versions. Rather than relying on the quality of a single test or code module, this "differential GAI" (D-GAI) approach promotes more reliable quality evaluation through version diversity. We introduce the Large-Scale Software Observatorium (LASSO), a platform that supports D-GAI by executing and analyzing large sets of code versions and tests. We discuss how LASSO enables rigorous evaluation of GAI-generated artifacts and propose its application in both software development and GAI research.