Large Language Models in Teaching and Learning: Reflections on Implementing an AI Chatbot in Higher Education

TL;DR

This paper explores the implementation of a retrieval-augmented large language model AI assistant in higher education, evaluating its impact on student motivation, response quality, and academic performance through mixed-methods experiments.

Contribution

It demonstrates a practical approach to integrating LLMs in university courses and assesses their pedagogical effectiveness and challenges in a real educational setting.

Findings

Students found the LLM engaging and motivating.

AI responses were comparable in quality to human-led exercises.

The LLM influenced students' academic performance positively.

Abstract

The landscape of education is changing rapidly, shaped by emerging pedagogical approaches, technological innovations such as artificial intelligence (AI), and evolving societal expectations, all of which demand thorough evaluation of new educational tools. Although large language models (LLMs) present substantial opportunities especially in Higher Education, their propensity to generate hallucinations and their limited specialized knowledge may introduce significant risks. This study aims to address these risks by examining the practical implementation of an LLM-enhanced assistant in a university level course. We implemented a generative AI assistant grounded in a retrieval-augmented generation (RAG) model to replicate a previously teacher-led, time-intensive exercise. To assess the effectiveness of the LLM, we conducted three separate experiments through iterative mixed-methods approaches, including a crossover design. The resulting data address central research questions related to student motivation, perceived differences between engaging with the LLM versus a human teacher, the quality of AI-generated responses, and the impact of the LLM on students' academic performance. The results offer direct insights into students' views and the pedagogical feasibility of embedding LLMs into specialized courses. Finally, we discuss the main challenges, opportunities and future directions of LLMs in teaching and learning in Higher Education.