Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions

TL;DR

This paper explores the use of vision-language models to detect students' academic emotions from facial expressions in online learning, demonstrating moderate success and highlighting potential for practical emotion recognition applications.

Contribution

It investigates the application of zero-shot vision-language models for academic emotion detection, showing their capabilities and limitations without requiring fine-tuning.

Findings

Models perform moderately in recognizing facial expressions.

Qwen2.5-VL-7B-Instruct outperforms Llama-3.2-11B-Vision-Instruct.

Models excel at identifying happy emotions but struggle with distracted behavior.

Abstract

Students' academic emotions significantly influence their social behavior and learning performance. Traditional approaches to automatically and accurately analyze these emotions have predominantly relied on supervised machine learning algorithms. However, these models often struggle to generalize across different contexts, necessitating repeated cycles of data collection, annotation, and training. The emergence of Vision-Language Models (VLMs) offers a promising alternative, enabling generalization across visual recognition tasks through zero-shot prompting without requiring fine-tuning. This study investigates the potential of VLMs to analyze students' academic emotions via facial expressions in an online learning environment. We employed two VLMs, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, to analyze 5,000 images depicting confused, distracted, happy, neutral, and tired expressions using zero-shot prompting. Preliminary results indicate that both models demonstrate moderate performance in academic facial expression recognition, with Qwen2.5-VL-7B-Instruct outperforming Llama-3.2-11B-Vision-Instruct. Notably, both models excel in identifying students' happy emotions but fail to detect distracted behavior. Additionally, Qwen2.5-VL-7B-Instruct exhibits relatively high performance in recognizing students' confused expressions, highlighting its potential for practical applications in identifying content that causes student confusion.