LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives

TL;DR

This study compares Dutch-specific LLMs and traditional sentiment tools in analyzing emotional valence in spontaneous narratives, revealing traditional methods outperform LLMs in low-resource language contexts.

Contribution

It provides an empirical evaluation of LLMs versus traditional tools for sentiment analysis in Dutch, highlighting the limitations of current LLM fine-tuning for emotional nuance detection.

Findings

Pattern outperformed LLMs in valence prediction

LLMs underperformed despite architectural advancements

Traditional tools are more effective for low-resource language sentiment analysis

Abstract

Understanding emotional nuances in everyday language is crucial for computational linguistics and emotion research. While traditional lexicon-based tools like LIWC and Pattern have served as foundational instruments, Large Language Models (LLMs) promise enhanced context understanding. We evaluated three Dutch-specific LLMs (ChocoLlama-8B-Instruct, Reynaerde-7B-chat, and GEITje-7B-ultra) against LIWC and Pattern for valence prediction in Flemish, a low-resource language variant. Our dataset comprised approximately 25000 spontaneous textual responses from 102 Dutch-speaking participants, each providing narratives about their current experiences with self-assessed valence ratings (-50 to +50). Surprisingly, despite architectural advancements, the Dutch-tuned LLMs underperformed compared to traditional methods, with Pattern showing superior performance. These findings challenge assumptions about LLM superiority in sentiment analysis tasks and highlight the complexity of capturing emotional valence in spontaneous, real-world narratives. Our results underscore the need for developing culturally and linguistically tailored evaluation frameworks for low-resource language variants, while questioning whether current LLM fine-tuning approaches adequately address the nuanced emotional expressions found in everyday language use.