OpenR: An Open Source Framework for Advanced Reasoning with Large Language Models

TL;DR

OpenR is an open-source framework that integrates data acquisition, reinforcement learning, and non-autoregressive decoding to enhance reasoning in large language models, inspired by OpenAI's o1 model, and demonstrated with improved performance on the MATH dataset.

Contribution

OpenR is the first open-source platform to implement core techniques of OpenAI's o1 model, advancing LLM reasoning through integrated reinforcement learning and process supervision.

Findings

Substantial reasoning performance improvements on the MATH dataset.

Effective use of test-time compute and reinforcement learning for reasoning.

OpenR code and models are publicly available for community development.

Abstract

In this technical report, we introduce OpenR, an open-source framework designed to integrate key components for enhancing the reasoning capabilities of large language models (LLMs). OpenR unifies data acquisition, reinforcement learning training (both online and offline), and non-autoregressive decoding into a cohesive software platform. Our goal is to establish an open-source platform and community to accelerate the development of LLM reasoning. Inspired by the success of OpenAI's o1 model, which demonstrated improved reasoning abilities through step-by-step reasoning and reinforcement learning, OpenR integrates test-time compute, reinforcement learning, and process supervision to improve reasoning in LLMs. Our work is the first to provide an open-source framework that explores the core techniques of OpenAI's o1 model with reinforcement learning, achieving advanced reasoning capabilities beyond traditional autoregressive methods. We demonstrate the efficacy of OpenR by evaluating it on the MATH dataset, utilising publicly available data and search methods. Our initial experiments confirm substantial gains, with relative improvements in reasoning and performance driven by test-time computation and reinforcement learning through process reward models. The OpenR framework, including code, models, and datasets, is accessible at https://openreasoner.github.io.