A systematic assessment of Large Language Models for constructing two-level fractional factorial designs

TL;DR

This paper systematically evaluates the ability of Large Language Models like GPT and Gemini to construct two-level fractional factorial designs, comparing their outputs to traditional designs based on resolution and aberration criteria.

Contribution

It introduces a novel assessment of LLMs for design construction, demonstrating their effectiveness in creating optimal factorial designs with limited runs and factors.

Findings

LLMs can effectively generate optimal designs for up to eight factors.

Designs produced by LLMs meet resolution and aberration standards.

The study provides a benchmark for LLM capabilities in experimental design.

Abstract

Two-level fractional factorial designs permit the study multiple factors using a limited number of runs. Traditionally, these designs are obtained from catalogs available in standard textbooks or statistical software. However, modern Large Language Models (LLMs) can now produce two-level fractional factorial designs, but the quality of these designs has not been previously assessed. In this paper, we perform a systematic evaluation of two popular classes of LLMs, namely GPT and Gemini models, to construct two-level fractional factorial designs with 8, 16, and 32 runs, and 4 to 26 factors. To this end, we use prompting techniques to develop a high-quality set of design construction tasks for the LLMs. We compare the designs obtained by the LLMs with the best-known designs in terms of resolution and minimum aberration criteria. We show that the LLMs can effectively construct optimal 8-, 16-, and 32-run designs with up to eight factors.