LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning

TL;DR

This paper proposes a novel collaborative optimization framework combining gradient descent and LLM-based inference, demonstrating improved prompt tuning performance through their complementary strengths.

Contribution

It introduces a new method that alternates between gradient-based and LLM-based optimization, leveraging their synergy for better prompt tuning results.

Findings

Combined optimization outperforms individual methods on various tasks.

LLMs effectively generate improved solutions based on gradient trajectories.

Synergistic approach yields consistent performance gains.

Abstract

Mastering a skill generally relies on both hands-on experience from doers and insightful, high-level guidance by mentors. Will this strategy also work well for solving complex non-convex optimization problems? Here, a common gradient-based optimizer acts like a disciplined doer, making locally optimal updates at each step. Large Language Models (LLMs) can also search for better solutions by inferring from natural language instructions, akin to a high-level mentor. In this paper, we show that these two participators are complementary to each other and can effectively collaborate as a combined optimization framework. The collaborative optimization is achieved by alternating between the gradient-based and LLM-based optimizers. We instruct LLMs to generate possibly improved solutions by taking parameter trajectories recorded during the previous stage of gradient-based optimization into account. Inferred results of LLMs are used as restarting points for the next stage of gradient optimization. We verify the effectiveness of this optimization framework on prompt tuning. By leveraging both the locally rigorous gradient-based optimizer and the high-level deductive LLM-based optimizer, the combined optimization method consistently yields improvements over competitive baselines on a variety of tasks. Our results demonstrate the synergistic effect of conventional gradient-based optimization and the inference ability of LLMs. The code is released at https://github.com/guozix/LLM-catalyst.