Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

摘要翻译：

数学推理因其复杂且结构化的特性，对语言模型而言是一项极具挑战性的任务。本文中，我们推出DeepSeekMath 7B模型：以DeepSeek-Coder-Base-v1.5 7B为基础，使用从Common Crawl中筛选出的1200亿数学相关token，搭配自然语言与代码数据完成继续预训练。

DeepSeekMath 7B在竞赛级MATH基准测试中取得了51.7%的亮眼成绩，全程不依赖外部工具包与多输出投票技术，性能已接近Gemini-Ultra和GPT-4的水平。对该模型输出的64条推理样本做自一致性校验后，其在MATH基准上的准确率进一步达到60.9%。

DeepSeekMath的数学推理能力来源于两大核心支撑：其一，我们通过精心设计的数据筛选流水线，充分挖掘了全网公开网页数据的巨大潜力；其二，我们提出了群组相对策略优化（GRPO）算法——作为近端策略优化（PPO）的改进变体，它在大幅提升模型数学推理能力的同时，还优化了传统PPO的内存占用问题。

关键术语说明：

  • ‌MATH基准‌：由斯坦福大学发布的高中数学竞赛级推理测试集，包含代数、几何、数论等多类复杂数学题，是行业公认的高难度数学推理评测标准

  • ‌自一致性（Self-consistency）‌：大模型推理增强技术，通过对同一问题生成多条不同推理路径，选择出现次数最多的答案作为最终结果，大幅提升复杂推理任务的准确率

  • ‌Common Crawl‌：公开的全网网页海量数据集，是大模型预训练语料的核心来源之一