FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring

TL;DR

This paper introduces FEAT, a cost-effective framework for generating high-quality teacher feedback datasets for English AI tutoring by combining manual and AI-generated data, improving feedback quality efficiently.

Contribution

The study presents a novel framework that combines manual and AI-generated feedback to create scalable, high-quality datasets for AI tutoring systems, reducing costs.

Findings

Incorporating 5-10% manual feedback improves dataset quality.

The combined datasets outperform purely manual datasets.

Cost-effective feedback generation enhances AI tutoring performance.

Abstract

In English education tutoring, teacher feedback is essential for guiding students. Recently, AI-based tutoring systems have emerged to assist teachers; however, these systems require high-quality and large-scale teacher feedback data, which is both time-consuming and costly to generate manually. In this study, we propose FEAT, a cost-effective framework for generating teacher feedback, and have constructed three complementary datasets: (1) DIRECT-Manual (DM), where both humans and large language models (LLMs) collaboratively generate high-quality teacher feedback, albeit at a higher cost; (2) DIRECT-Generated (DG), an LLM-only generated, cost-effective dataset with lower quality;, and (3) DIRECT-Augmented (DA), primarily based on DG with a small portion of DM added to enhance quality while maintaining cost-efficiency. Experimental results showed that incorporating a small portion of DM (5-10%) into DG leads to superior performance compared to using 100% DM alone.