AI Conversational Tutors in Foreign Language Learning: A Mixed-Methods Evaluation Study

TL;DR

This study evaluates state-of-the-art AI conversational tutors for foreign language learning, focusing on user experience, conversation quality, and privacy concerns to guide future system design and assessment criteria.

Contribution

It provides a comprehensive mixed-methods evaluation of AI language tutors, highlighting key quality metrics and privacy considerations for improving future tools.

Findings

Identified key factors affecting user experience and conversation quality.

Established criteria for assessing AI tutor effectiveness.

Highlighted privacy and data security concerns in AI tutoring systems.

Abstract

This paper focuses on AI tutors in foreign language learning, a field of application of AI tutors with great development, especially during the last years, when great advances in natural language understanding and processing in real time, have been achieved. These tutors attempt to address needs for improving language skills (speaking, or communicative competence, understanding). In this paper, a mixed-methos empirical study on the use of different kinds of state-of-the-art AI tutors for language learning is reported. This study involves a user experience evaluation of typical such tools, with special focus in their conversation functionality and an evaluation of their quality, based on chat transcripts. This study can help establish criteria for assessing the quality of such systems and inform the design of future tools, including concerns about data privacy and secure handling of learner information.