JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding

TL;DR

This paper introduces JiuZhang, a Chinese pre-trained language model tailored for understanding and solving mathematical problems, utilizing a curriculum pre-training approach with logic-based tasks to enhance mathematical reasoning.

Contribution

The paper presents the first Chinese mathematical pre-trained language model with a novel curriculum pre-training strategy incorporating logic-based tasks for improved mathematical problem understanding.

Findings

Outperforms baseline models on nine math-related tasks

Effective in both offline evaluation and online A/B testing

Demonstrates significant improvement in mathematical problem understanding

Abstract

This paper aims to advance the mathematical intelligence of machines by presenting the first Chinese mathematical pre-trained language model~(PLM) for effectively understanding and representing mathematical problems. Unlike other standard NLP tasks, mathematical texts are difficult to understand, since they involve mathematical terminology, symbols and formulas in the problem statement. Typically, it requires complex mathematical logic and background knowledge for solving mathematical problems. Considering the complex nature of mathematical texts, we design a novel curriculum pre-training approach for improving the learning of mathematical PLMs, consisting of both basic and advanced courses. Specially, we first perform token-level pre-training based on a position-biased masking strategy, and then design logic-based pre-training tasks that aim to recover the shuffled sentences and formulas, respectively. Finally, we introduce a more difficult pre-training task that enforces the PLM to detect and correct the errors in its generated solutions. We conduct extensive experiments on offline evaluation (including nine math-related tasks) and online $A/B$ test. Experimental results demonstrate the effectiveness of our approach compared with a number of competitive baselines. Our code is available at: \textcolor{blue}{\url{https://github.com/RUCAIBox/JiuZhang}}.