Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring

TL;DR

This study examines the ability of system prompts to control language difficulty in LLMs for Spanish tutoring, revealing that prompt effectiveness diminishes over time due to alignment drift, impacting adaptive learning applications.

Contribution

It introduces the concept of alignment drift in LLMs during interactive language tutoring and evaluates the limitations of CEFR-based prompting for sustained proficiency alignment.

Findings

Prompting can influence output difficulty but is unreliable over long interactions.

Alignment drift causes models to deviate from targeted proficiency levels.

The method enables scalable, low-cost evaluation of LLMs for language education.

Abstract

This paper investigates the potentials of Large Language Models (LLMs) as adaptive tutors in the context of second-language learning. In particular, we evaluate whether system prompting can reliably constrain LLMs to generate only text appropriate to the student's competence level. We simulate full teacher-student dialogues in Spanish using instruction-tuned, open-source LLMs ranging in size from 7B to 12B parameters. Dialogues are generated by having an LLM alternate between tutor and student roles with separate chat histories. The output from the tutor model is then used to evaluate the effectiveness of CEFR-based prompting to control text difficulty across three proficiency levels (A1, B1, C1). Our findings suggest that while system prompting can be used to constrain model outputs, prompting alone is too brittle for sustained, long-term interactional contexts - a phenomenon we term alignment drift. Our results provide insights into the feasibility of LLMs for personalized, proficiency-aligned adaptive tutors and provide a scalable method for low-cost evaluation of model performance without human participants.