Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length

TL;DR

This paper explores using Proximal Policy Optimization (PPO) to manipulate the tokenizer length of language model outputs, demonstrating its effectiveness and training facilitation in a novel task involving reward models.

Contribution

It introduces a new task to validate PPO's ability to control output tokenizer length and shows PPO's effectiveness and improved training stability in this context.

Findings

PPO can effectively manipulate output tokenizer length.

Training is facilitated when the reward model influence is excluded.

The task validates PPO's potential in output control for language models.

Abstract

The Reinforcement Learning from Human Feedback (RLHF) plays a pivotal role in shaping the impact of large language models (LLMs), contributing significantly to controlling output toxicity and selecting output styles, particularly as LLMs often harbor misleading content, highlighting the urgency to align them with human values for secure AI systems. The RLHF, characterized by complexity, instability, and sensitivity to hyperparameters, makes the evaluation of the reward model for complex tasks challenging, thereby further complicating the use of Proximal Policy Optimization (PPO). In this paper, we introduce a simple task designed to employ Gloden as a reward model that validates the effectiveness of PPO and inspires it, primarily explaining the task of utilizing PPO to manipulate the tokenizer length of the output generated by the model. Experiments confirm that PPO is not only effective in manipulating the output tokenizer length to a certain extent in this type of task but also exhibits facilitated training once the influence of the reward model effect is excluded, making it an exciting development.