Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications

TL;DR

This study evaluates the emotion recognition abilities of multimodal large language models, GPT-4o and Gemini 1.5 Pro, in processing Arabic children's storybook images, revealing performance gaps and cultural understanding limitations.

Contribution

It provides a comparative analysis of two advanced multimodal LLMs in Arabic emotion recognition, highlighting their strengths, weaknesses, and the need for culturally sensitive AI training.

Findings

GPT-4o outperformed Gemini in all prompting conditions.

Highest macro F1-score of 59% achieved with GPT-4o using chain-of-thought prompting.

Systematic misclassification patterns, especially valence inversions, were identified.

Abstract

Emotion recognition capabilities in multimodal AI systems are crucial for developing culturally responsive educational technologies, yet remain underexplored for Arabic language contexts where culturally appropriate learning tools are critically needed. This study evaluates the emotion recognition performance of two advanced multimodal large language models, GPT-4o and Gemini 1.5 Pro, when processing Arabic children's storybook illustrations. We assessed both models across three prompting strategies (zero-shot, few-shot, and chain-of-thought) using 75 images from seven Arabic storybooks, comparing model predictions with human annotations based on Plutchik's emotional framework. GPT-4o consistently outperformed Gemini across all conditions, achieving the highest macro F1-score of 59% with chain-of-thought prompting compared to Gemini's best performance of 43%. Error analysis revealed systematic misclassification patterns, with valence inversions accounting for 60.7% of errors, while both models struggled with culturally nuanced emotions and ambiguous narrative contexts. These findings highlight fundamental limitations in current models' cultural understanding and emphasize the need for culturally sensitive training approaches to develop effective emotion-aware educational technologies for Arabic-speaking learners.