Learning Guarantee of Reward Modeling Using Deep Neural Networks

TL;DR

This paper provides a theoretical analysis of reward modeling with deep neural networks, establishing regret bounds that depend on network architecture and emphasizing the importance of clear human beliefs for efficient learning.

Contribution

It introduces a non-asymptotic regret bound for deep reward estimators and a margin-type condition that improves bounds and explains empirical success of reinforcement learning from human feedback.

Findings

Regret bounds depend explicitly on network architecture.

Margin condition improves regret bounds and highlights human belief importance.

High-quality pairwise data enhances learning efficiency.

Abstract

In this work, we study the learning theory of reward modeling with pairwise comparison data using deep neural networks. We establish a novel non-asymptotic regret bound for deep reward estimators in a non-parametric setting, which depends explicitly on the network architecture. Furthermore, to underscore the critical importance of clear human beliefs, we introduce a margin-type condition that assumes the conditional winning probability of the optimal action in pairwise comparisons is significantly distanced from 1/2. This condition enables a sharper regret bound, which substantiates the empirical efficiency of Reinforcement Learning from Human Feedback and highlights clear human beliefs in its success. Notably, this improvement stems from high-quality pairwise comparison data implied by the margin-type condition, is independent of the specific estimators used, and thus applies to various learning algorithms and models.