DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

TL;DR

DeepSeekMath 7B significantly advances mathematical reasoning in language models by leveraging extensive web data and a novel optimization technique, achieving near state-of-the-art performance on the MATH benchmark without external tools.

Contribution

The paper introduces DeepSeekMath 7B, a new pre-trained model that improves mathematical reasoning through targeted data selection and a novel optimization method called GRPO.

Findings

Achieves 51.7% on MATH benchmark without external tools.

Self-consistency sampling improves performance to 60.9%.

Introduces Group Relative Policy Optimization (GRPO) for better reasoning and memory efficiency.

Abstract

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.