MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors

TL;DR

This paper introduces MSA-MathEval, a unified instruction-tuned language model with a disagreement-aware ensemble approach, achieving top performance in multi-dimensional evaluation of LLMs as math tutors.

Contribution

It presents a scalable, task-agnostic training pipeline and a disagreement-aware inference strategy for robust multi-dimensional LLM evaluation.

Findings

Achieved 1st place in Providing Guidance

Ranked 3rd in Actionability

Placed 4th in Mistake Identification and Mistake Location

Abstract

We present MSA-MathEval, our submission to the BEA 2025 Shared Task on evaluating AI tutor responses across four instructional dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. Our approach uses a unified training pipeline to fine-tune a single instruction-tuned language model across all tracks, without any task-specific architectural changes. To improve prediction reliability, we introduce a disagreement-aware ensemble inference strategy that enhances coverage of minority labels. Our system achieves strong performance across all tracks, ranking 1st in Providing Guidance, 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location. These results demonstrate the effectiveness of scalable instruction tuning and disagreement-driven modeling for robust, multi-dimensional evaluation of LLMs as educational tutors.