Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement

TL;DR

This paper introduces a reinforcement learning-based prompt refinement framework that improves large language models' response quality and robustness against harmful jailbreak prompts by enhancing query quality before input.

Contribution

It presents a transferable, pluggable prompt refinement model trained with a novel reinforcement learning approach to boost LLMs' honesty, harmlessness, and resistance to adversarial prompts.

Findings

Enhanced response quality in LLMs

Improved robustness against jailbreak attacks

Effective reinforcement learning training method

Abstract

The capacity of large language models (LLMs) to generate honest, harmless, and helpful responses heavily relies on the quality of user prompts. However, these prompts often tend to be brief and vague, thereby significantly limiting the full potential of LLMs. Moreover, harmful prompts can be meticulously crafted and manipulated by adversaries to jailbreak LLMs, inducing them to produce potentially toxic content. To enhance the capabilities of LLMs while maintaining strong robustness against harmful jailbreak inputs, this study proposes a transferable and pluggable framework that refines user prompts before they are input into LLMs. This strategy improves the quality of the queries, empowering LLMs to generate more truthful, benign and useful responses. Specifically, a lightweight query refinement model is introduced and trained using a specially designed reinforcement learning approach that incorporates multiple objectives to enhance particular capabilities of LLMs. Extensive experiments demonstrate that the refinement model not only improves the quality of responses but also strengthens their robustness against jailbreak attacks. Code is available at: https://github.com/Huangzisu/query-refinement .