NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors

TL;DR

This paper introduces a retrieval-augmented prompting system using GPT-4 for mistake identification in AI tutors, combining multiple models and retrieval techniques to improve pedagogical feedback accuracy.

Contribution

It presents a novel retrieval-augmented prompting approach that enhances mistake detection in AI tutoring systems, outperforming baseline methods.

Findings

Retrieval-augmented prompting improves mistake identification accuracy.

Combining multiple models yields better performance than individual approaches.

The system provides interpretable, schema-guided predictions.

Abstract

This paper presents our system for Track 1: Mistake Identification in the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors. The task involves evaluating whether a tutor's response correctly identifies a mistake in a student's mathematical reasoning. We explore four approaches: (1) an ensemble of machine learning models over pooled token embeddings from multiple pretrained language models (LMs); (2) a frozen sentence-transformer using [CLS] embeddings with an MLP classifier; (3) a history-aware model with multi-head attention between token-level history and response embeddings; and (4) a retrieval-augmented few-shot prompting system with a large language model (LLM) i.e. GPT 4o. Our final system retrieves semantically similar examples, constructs structured prompts, and uses schema-guided output parsing to produce interpretable predictions. It outperforms all baselines, demonstrating the effectiveness of combining example-driven prompting with LLM reasoning for pedagogical feedback assessment. Our code is available at https://github.com/NaumanNaeem/BEA_2025.