Understanding the Challenges in Iterative Generative Optimization with LLMs

TL;DR

This paper investigates the practical challenges of iterative generative optimization with large language models, highlighting how design choices impact success and providing guidance for better setup across applications.

Contribution

It identifies key factors influencing generative optimization success and offers practical recommendations to improve its robustness and applicability across domains.

Findings

Starting artifacts influence solution reachability.

Truncated traces can still enhance performance.

Larger minibatches do not always improve generalization.

Abstract

Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.