Using Code Generation to Solve Open Instances of Combinatorial Design Problems

TL;DR

This paper introduces CPro1, a protocol leveraging large language models to generate code that constructs combinatorial designs, successfully solving several open instances by automating exploration of various methods.

Contribution

The paper presents a novel protocol that uses LLMs to generate and test code for solving open combinatorial design problems, automating the discovery of effective construction strategies.

Findings

Successfully solved open instances in 6 design types

Automated exploration of multiple construction methods

Demonstrated effectiveness of LLM-generated code in combinatorial design

Abstract

The Handbook of Combinatorial Designs catalogs many types of combinatorial designs, together with lists of open instances for which existence has not yet been determined. We develop a constructive protocol CPro1, which uses Large Language Models (LLMs) to generate code that constructs combinatorial designs and resolves some of these open instances. The protocol starts from a definition of a particular type of design, and a verifier that reliably confirms whether a proposed design is valid. The LLM selects strategies and implements them in code, and scaffolding provides automated hyperparameter tuning and execution feedback using the verifier. Most generated code fails, but by generating many candidates, the protocol automates exploration of a variety of standard methods (e.g. simulated annealing, genetic algorithms) and experimentation with variations (e.g. cost functions) to find successful approaches. Testing on 16 different types of designs, CPro1 constructs solutions to open instances for 6 of them: Symmetric and Skew Weighing Matrices, Equidistant Permutation Arrays, Packing Arrays, Balanced Ternary Designs, and Florentine Rectangles.