Automated Design Optimization via Strategic Search with Large Language Models

TL;DR

This paper introduces AUTO, an LLM-based framework for design optimization that uses strategic reasoning to efficiently explore complex, ill-defined search spaces, demonstrating competitive results and cost savings in GPU code optimization.

Contribution

The paper presents AUTO, a novel LLM-driven approach that treats design optimization as a gradient-free search, integrating strategic reasoning and collaborative agents for improved efficiency.

Findings

Achieves 50-70% search efficiency compared to Bayesian optimization.

Completes optimizations in about 8 hours at a lower cost than manual development.

Generates solutions competitive with expert implementations in GPU code optimization.

Abstract

Traditional optimization methods excel in well-defined search spaces but struggle with design problems where transformations and design parameters are difficult to define. Large language models (LLMs) offer a promising alternative by dynamically interpreting design spaces and leveraging encoded domain knowledge. To this end, we introduce AUTO, an LLM agent framework that treats design optimization as a gradient-free search problem guided by strategic LLM reasoning. The framework employs two collaborative agents: a Strategist that selects between exploration and exploitation strategies, and an Implementor that executes detailed designs. Applied to GPU code optimization -- a domain critical to fields from machine learning to scientific computing -- AUTO generates solutions competitive with expert implementations for chemical kinetics integration and dense matrix multiplication. The framework achieves 50-70% search efficiency relative to Bayesian optimization methodologies. It completes optimizations in approximately 8 hours at an estimated cost of up to \$159 per run, compared to an estimated cost of up to \$480 with median-wage software developers. These findings open the door to automating design optimization in ill-defined search spaces with limited prior information.