Scaling Textual Gradients via Sampling-Based Momentum

TL;DR

This paper introduces TSGD-M, a sampling-based momentum method for scaling textual gradient optimization in prompt engineering, addressing challenges of data scaling and stability in large language model prompts.

Contribution

It proposes a novel sampling-based momentum approach with Gumbel-Top-k sampling, improving scalability and stability in textual gradient descent for prompt optimization.

Findings

TSGD-M achieves consistent improvements across 5 benchmarks.

Gumbel-Top-k sampling balances exploration and exploitation effectively.

The method integrates seamlessly with existing prompt optimization frameworks.

Abstract

LLM-based prompt optimization, that uses LLM-provided "textual gradients" (feedback) to refine prompts, has emerged an effective method for automatic prompt engineering. However, its scalability and stability are unclear when using more data in training. We systematically investigate the potential and challenges of scaling training data in textual gradient descent. We show that naively scaling training examples is infeasible due to both explicit context-length limits and an implicit context wall, where long-context degradation yields diminishing returns. Inspired by prior wisdom in stochastic gradient descent, we propose Textual Stochastic Gradient Descent with Momentum (TSGD-M), which reweights updates through momentum sampling, using bootstrapped minibatch validation accuracy as importance weights over historical prompts. We introduce Gumbel-Top-$k$ sampling for prompt generation, balancing exploration--exploitation and improving sampling efficiency while maintaining a low-variance running mean estimator. TSGD-M integrates seamlessly into existing prompt optimization frameworks, including TextGrad, DSPy-COPRO, and AdalFlow, and achieves consistent gains across 5 benchmarks.