Evaluation of mathematical questioning strategies using data collected through weak supervision

TL;DR

This paper introduces an AI-based classroom simulator that uses weak supervision and human-in-the-loop methods to generate high-quality training data for mathematical questioning strategies, aiming to improve teacher training and research.

Contribution

It presents a novel high-fidelity simulator and a data collection approach leveraging uncertainty quantification and human-in-the-loop techniques for mathematical questioning scenarios.

Findings

Effective data collection via human-in-the-loop enhances training data quality.

Uncertainty quantification aids in evaluating conversational agent usability.

The approach reduces costs and improves practicality of scenario development.

Abstract

A large body of research demonstrates how teachers' questioning strategies can improve student learning outcomes. However, developing new scenarios is challenging because of the lack of training data for a specific scenario and the costs associated with labeling. This paper presents a high-fidelity, AI-based classroom simulator to help teachers rehearse research-based mathematical questioning skills. Using a human-in-the-loop approach, we collected a high-quality training dataset for a mathematical questioning scenario. Using recent advances in uncertainty quantification, we evaluated our conversational agent for usability and analyzed the practicality of incorporating a human-in-the-loop approach for data collection and system evaluation for a mathematical questioning scenario.