LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms

TL;DR

This paper introduces LLM-SEM, a novel sentiment-based metric leveraging large language models to accurately and scalably measure student engagement in e-learning platforms through metadata and comment analysis.

Contribution

It presents a new approach combining metadata and sentiment analysis with LLMs to improve engagement measurement in e-learning environments.

Findings

LLM-SEM effectively captures engagement at course and lesson levels.

Fine-tuning LLMs improves sentiment prediction accuracy.

The method is scalable and outperforms traditional engagement metrics.

Abstract

Current methods for analyzing student engagement in e-learning platforms, including automated systems, often struggle with challenges such as handling fuzzy sentiment in text comments and relying on limited metadata. Traditional approaches, such as surveys and questionnaires, also face issues like small sample sizes and scalability. In this paper, we introduce LLM-SEM (Language Model-Based Student Engagement Metric), a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement. By utilizing recent Large Language Models (LLMs), we generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes. Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels. Extensive experiments were conducted to evaluate various LLM models, demonstrating the effectiveness of LLM-SEM in providing a scalable and accurate measure of student engagement. We fine-tuned TXLM-RoBERTa using human-annotated sentiment datasets to enhance prediction accuracy and utilized LLama 3B, and Gemma 9B from Ollama.