GFlowPO: Generative Flow Network as a Language Model Prompt Optimizer

TL;DR

GFlowPO introduces a probabilistic framework using Generative Flow Networks to efficiently optimize prompts for language models, improving sample efficiency and performance across various NLP tasks.

Contribution

It presents a novel prompt optimization method that combines GFlowNets with a dynamic memory update mechanism for better exploration and exploitation.

Findings

Outperforms recent prompt optimization baselines.

Achieves higher rewards in few-shot classification and QA.

Demonstrates sample-efficient exploration with replay-based training.

Abstract

Finding effective prompts for language models (LMs) is critical yet notoriously difficult: the prompt space is combinatorially large, rewards are sparse due to expensive target-LM evaluation. Yet, existing RL-based prompt optimizers often rely on on-policy updates and a meta-prompt sampled from a fixed distribution, leading to poor sample efficiency. We propose GFlowPO, a probabilistic prompt optimization framework that casts prompt search as a posterior inference problem over latent prompts regularized by a meta-prompted reference-LM prior. In the first step, we fine-tune a lightweight prompt-LM with an off-policy Generative Flow Network (GFlowNet) objective, using a replay-based training policy that reuses past prompt evaluations to enable sample-efficient exploration. In the second step, we introduce Dynamic Memory Update (DMU), a training-free mechanism that updates the meta-prompt by injecting both (i) diverse prompts from a replay buffer and (ii) top-performing prompts from a small priority queue, thereby progressively concentrating the search process on high-reward regions. Across few-shot text classification, instruction induction benchmarks, and question answering tasks, GFlowPO consistently outperforms recent discrete prompt optimization baselines.