The Real, the Better: Aligning Large Language Models with Online Human Behaviors

TL;DR

This paper introduces RLHB, a novel alignment framework for large language models that leverages real online human behaviors through a generative adversarial approach, improving alignment with diverse human preferences.

Contribution

The paper proposes RLHB, a new online alignment method using adversarial training and behavior modeling to better align LLMs with real human behaviors.

Findings

Effective alignment with online human behaviors confirmed by evaluations

Improved response quality and alignment accuracy

Sustainable online adaptation demonstrated

Abstract

Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.