Reward Dropout Improves Control: Bi-objective Perspective on Reinforced LM

TL;DR

This paper presents a bi-objective perspective on Reinforced Language Models, establishing their theoretical foundation as Pareto optimization problems and introducing Reward Dropout to enhance their performance across multiple benchmarks.

Contribution

It introduces Reward Dropout, a novel method that improves bi-objective optimization in RLMs, supported by theoretical analysis and empirical validation.

Findings

Reward Dropout consistently improves RLM performance.

Theoretical foundations of RLM as Pareto optimization are established.

Empirical results show significant gains across benchmarks.

Abstract

We study the theoretical aspects of Reinforced Language Models (RLMs) from a bi-objective optimization perspective. Specifically, we consider the RLMs as a Pareto optimization problem that maximizes the two conflicting objectives, i.e., reward objective and likelihood objectives, simultaneously. Our main contribution consists of three parts. First, we establish the theoretical foundations of RLM as a Pareto optimization problem by presenting Reward Upper BOund (RUBO) and Pareto optimality. Our theoretical outcomes are supported by not only deductive proofs but also empirical results. Second, we propose Reward Dropout, a simple yet powerful method that guarantees to improve a bi-objective optimization of RLM. Lastly, we demonstrate that the Reward Dropout is consistently effective across five benchmark datasets and four benchmark LLMs, meaning that the Reward Dropout significantly improves the optimization performance of RLMs.