ChatGPT as a Math Questioner? Evaluating ChatGPT on Generating Pre-university Math Questions

TL;DR

This paper evaluates ChatGPT's ability to generate pre-university math questions, analyzing its performance in different settings and introducing a new dataset from curriculum crawling to assess its potential as an educational question generator.

Contribution

It is the first comprehensive analysis of ChatGPT's effectiveness in generating math questions across various educational levels and settings, supported by a new curriculum dataset.

Findings

ChatGPT performs well in context-aware question answering benchmarks.

ChatGPT can generate diverse math questions for different curriculum levels.

The study introduces TopicMath, a new dataset from 121 topics and 428 lessons.

Abstract

Mathematical questioning is crucial for assessing students problem-solving skills. Since manually creating such questions requires substantial effort, automatic methods have been explored. Existing state-of-the-art models rely on fine-tuning strategies and struggle to generate questions that heavily involve multiple steps of logical and arithmetic reasoning. Meanwhile, large language models(LLMs) such as ChatGPT have excelled in many NLP tasks involving logical and arithmetic reasoning. Nonetheless, their applications in generating educational questions are underutilized, especially in the field of mathematics. To bridge this gap, we take the first step to conduct an in-depth analysis of ChatGPT in generating pre-university math questions. Our analysis is categorized into two main settings: context-aware and context-unaware. In the context-aware setting, we evaluate ChatGPT on existing math question-answering benchmarks covering elementary, secondary, and ternary classes. In the context-unaware setting, we evaluate ChatGPT in generating math questions for each lesson from pre-university math curriculums that we crawl. Our crawling results in TopicMath, a comprehensive and novel collection of pre-university math curriculums collected from 121 math topics and 428 lessons from elementary, secondary, and tertiary classes. Through this analysis, we aim to provide insight into the potential of ChatGPT as a math questioner.