Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training

TL;DR

This paper introduces QuesCo, a contrastive pre-training method for mathematical question representation that leverages hierarchical knowledge and diverse augmentations to improve understanding and application in tasks like difficulty estimation.

Contribution

The paper presents a novel contrastive pre-training approach with hierarchical knowledge-aware ranking and dual-level question augmentations for better mathematical question understanding.

Findings

QuesCo outperforms baseline models on real-world datasets.

Hierarchical knowledge integration improves question similarity ranking.

Dual-level augmentations enhance model robustness and generalization.

Abstract

Understanding mathematical questions effectively is a crucial task, which can benefit many applications, such as difficulty estimation. Researchers have drawn much attention to designing pre-training models for question representations due to the scarcity of human annotations (e.g., labeling difficulty). However, unlike general free-format texts (e.g., user comments), mathematical questions are generally designed with explicit purposes and mathematical logic, and usually consist of more complex content, such as formulas, and related mathematical knowledge (e.g., Function). Therefore, the problem of holistically representing mathematical questions remains underexplored. To this end, in this paper, we propose a novel contrastive pre-training approach for mathematical question representations, namely QuesCo, which attempts to bring questions with more similar purposes closer. Specifically, we first design two-level question augmentations, including content-level and structure-level, which generate literally diverse question pairs with similar purposes. Then, to fully exploit hierarchical information of knowledge concepts, we propose a knowledge hierarchy-aware rank strategy (KHAR), which ranks the similarities between questions in a fine-grained manner. Next, we adopt a ranking contrastive learning task to optimize our model based on the augmented and ranked questions. We conduct extensive experiments on two real-world mathematical datasets. The experimental results demonstrate the effectiveness of our model.