Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models

TL;DR

Trinity-RFT is a versatile, modular framework that simplifies reinforcement fine-tuning of large language models, supporting various modes and applications with high efficiency.

Contribution

It introduces a unified, flexible platform for reinforcement fine-tuning of large language models, integrating multiple RFT modes and optimizing data pipelines.

Findings

Demonstrates high efficiency and robustness in diverse RFT scenarios.

Supports both on-policy and off-policy reinforcement learning.

Facilitates research and development of advanced RL paradigms.

Abstract

Trinity-RFT is a general-purpose, unified and easy-to-use framework designed for reinforcement fine-tuning (RFT) of large language models. It is built with a modular and decoupled design, consisting of (1) an RFT-core that unifies and generalizes synchronous/asynchronous, on-policy/off-policy, and online/offline modes of RFT; (2) seamless integration for agent-environment interaction with high efficiency and robustness; and (3) systematic data pipelines optimized for RFT. Trinity-RFT can be easily adapted for diverse application scenarios, and serves as a unified platform for development and research of advanced reinforcement learning paradigms at both macroscopic and microscopic levels. This technical report outlines the vision, features, design and implementations of Trinity-RFT, accompanied by extensive examples, applications and experiments that demonstrate its functionalities and user-friendliness.