G-Core: A Simple, Scalable and Balanced RLHF Trainer

TL;DR

G-Core is a new RLHF training framework that enhances scalability, flexibility, and efficiency for large language models by introducing parallel control and dynamic resource management, demonstrated on real-world applications.

Contribution

G-Core offers a novel parallel controller programming model and adaptive resource placement schema to improve RLHF training scalability and efficiency in complex, dynamic workflows.

Findings

Successfully trained models for WeChat features with large user base

Reduced hardware idle time and improved resource utilization

Enhanced scalability and robustness in real-world RLHF scenarios

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become an increasingly popular paradigm for training large language models (LLMs) and diffusion models. While existing RLHF training systems have enabled significant progress, they often face challenges in scaling to multi-modal and diffusion workflows and adapting to dynamic workloads. In particular, current approaches may encounter limitations in controller scalability, flexible resource placement, and efficient orchestration when handling complex RLHF pipelines, especially in scenarios involving dynamic sampling or generative reward modeling. In this paper, we present \textbf{G-Core}, a simple, scalable, and balanced RLHF training framework designed to address these challenges. G-Core introduces a parallel controller programming model, enabling flexible and efficient orchestration of complex RLHF workflows without the bottlenecks of a single centralized controller. Furthermore, we propose a dynamic placement schema that adaptively partitions resources and schedules workloads, significantly reducing hardware idle time and improving utilization, even under highly variable training conditions. G-Core has successfully trained models that support WeChat product features serving a large-scale user base, demonstrating its effectiveness and robustness in real-world scenarios. Our results show that G-Core advances the state of the art in RLHF training, providing a solid foundation for future research and deployment of large-scale, human-aligned models.