Learning to Reason: Training LLMs with GPT-OSS or DeepSeek R1 Reasoning Traces

TL;DR

This paper investigates how training medium-sized LLMs with reasoning traces generated by advanced models like DeepSeek-R1 and gpt-oss improves their ability to solve math problems, focusing on accuracy and efficiency.

Contribution

It compares the effectiveness of reasoning traces from DeepSeek-R1 and gpt-oss in enhancing medium-sized LLMs' math problem-solving performance.

Findings

DeepSeek-R1 traces lead to higher accuracy than gpt-oss traces.

Training with these traces improves inference efficiency.

The choice of reasoning trace source impacts model performance.

Abstract

Test-time scaling, which leverages additional computation during inference to improve model accuracy, has enabled a new class of Large Language Models (LLMs) that are able to reason through complex problems by understanding the goal, turning this goal into a plan, working through intermediate steps, and checking their own work before answering . Frontier large language models with reasoning capabilities, such as DeepSeek-R1 and OpenAI's gpt-oss, follow the same procedure when solving complex problems by generating intermediate reasoning traces before giving the final answer. Today, these models are being increasingly used to generate reasoning traces that serve as high-quality supervised data for post-training of small and medium-sized language models to teach reasoning capabilities without requiring expensive human curation. In this work, we compare the performance of medium-sized LLMs on Math problems after post-training on two kinds of reasoning traces. We compare the impact of reasoning traces generated by DeepSeek-R1 and gpt-oss LLMs in terms of accuracy and inference efficiency.