End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics

TL;DR

This paper presents an end-to-end evaluation of a spoken dialogue system designed to teach basic math concepts to young children, highlighting real-world deployment results and the impact of error propagation in multimodal interactions.

Contribution

It introduces a task-oriented SDS for early math education and explores improvements using MathBERT for natural language understanding.

Findings

Real-world deployment shows system effectiveness in early math learning.

Error propagation from ASR to dialogue management significantly affects performance.

Utilizing MathBERT enhances NLU capabilities in the SDS.

Abstract

The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for social-good opportunities with a broader positive impact. Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages. Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts. This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learning of basic math concepts for early childhood education. The system has been evaluated via real-world deployments at school while the students are practicing early math concepts with multimodal interactions. We discuss our efforts to improve the SDS pipeline built for math learning, for which we explore utilizing MathBERT representations for potential enhancement to the Natural Language Understanding (NLU) module. We perform an end-to-end evaluation using real-world deployment outputs from the Automatic Speech Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to understand how error propagation affects the overall performance in real-world scenarios.