Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning

TL;DR

Prompt-R1 is an end-to-end reinforcement learning framework enabling small LLMs to collaboratively generate prompts for large LLMs, improving problem-solving performance through multi-turn interactions.

Contribution

It introduces a novel collaborative prompting framework using reinforcement learning, enhancing large LLMs' effectiveness without user intervention.

Findings

Prompt-R1 outperforms baseline models on multiple datasets.

The framework effectively optimizes correctness, quality, and reasoning accuracy.

Supports both inference and training with various large-scale LLMs.

Abstract

Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.