How to Align Large Language Models for Teaching English? Designing and Developing LLM based-Chatbot for Teaching English Conversation in EFL, Findings and Limitations

TL;DR

This paper presents the design, development, and evaluation of an LLM-based chatbot aimed at teaching English conversation skills in EFL contexts, highlighting effective models, features, and future directions.

Contribution

It introduces a systematic approach to designing and refining an LLM chatbot for EFL, including evaluation methods and insights into system features and ethical considerations.

Findings

Identified effective LLM and prompt combinations for high-quality responses.

Highlighting the importance of feedback mechanisms and customizable AI personas.

Provided insights into desirable system features and ethical considerations.

Abstract

This study investigates the design, development, and evaluation of a Large Language Model (LLM)-based chatbot for teaching English conversations in an English as a Foreign Language (EFL) context. Employing the Design and Development Research (DDR), we analyzed needs, established design principles, and iteratively refined a chatbot through experimenting various LLMs and alignment methods. Through both quantitative and qualitative evaluations, we identified the most effective LLM and its prompt combination to generate high-quality, contextually appropriate responses. Interviews with teachers provided insights into desirable system features, potential educational applications, and ethical considerations in the development and deployment of the chatbots. The design iterations yielded the importance of feedback mechanisms and customizable AI personas. Future research should explore adaptive feedback strategies, collaborative approaches with various stakeholders, and the integration of insights from human-computer interaction (HCI) and user experience (UX) design. This study contributes to the growing body of research on applying LLMs in language education, providing insights and recommendations for the design, development, and evaluation of LLM-based chatbots for EFL conversation practice. As the field evolves, ongoing research and collaboration among educators, AI engineers, and other stakeholders will be essential to harness the potential of these technologies to enhance language learning experiences.