Are Lexicon-Based Tools Still the Gold Standard for Valence Analysis in Low-Resource Flemish?

TL;DR

This study evaluates the effectiveness of traditional lexicon-based tools versus Dutch-specific LLMs in capturing emotional valence in spontaneous Flemish language, finding current models insufficient and highlighting the need for culturally tailored solutions.

Contribution

It provides an empirical comparison of LLMs and traditional tools for valence analysis in low-resource Flemish, emphasizing the necessity for culturally adapted models.

Findings

Dutch LLMs underperform compared to traditional tools in real-world narratives.

Traditional lexicon-based tools like LIWC and Pattern remain more accurate for valence detection.

Highlighting the need for developing culturally and linguistically tailored models for emotion analysis.

Abstract

Understanding the nuances in everyday language is pivotal for advancements in computational linguistics & emotions research. Traditional lexicon-based tools such as LIWC and Pattern have long served as foundational instruments in this domain. LIWC is the most extensively validated word count based text analysis tool in the social sciences and Pattern is an open source Python library offering functionalities for NLP. However, everyday language is inherently spontaneous, richly expressive, & deeply context dependent. To explore the capabilities of LLMs in capturing the valences of daily narratives in Flemish, we first conducted a study involving approximately 25,000 textual responses from 102 Dutch-speaking participants. Each participant provided narratives prompted by the question, "What is happening right now and how do you feel about it?", accompanied by self-assessed valence ratings on a continuous scale from -50 to +50. We then assessed the performance of three Dutch-specific LLMs in predicting these valence scores, and compared their outputs to those generated by LIWC and Pattern. Our findings indicate that, despite advancements in LLM architectures, these Dutch tuned models currently fall short in accurately capturing the emotional valence present in spontaneous, real-world narratives. This study underscores the imperative for developing culturally and linguistically tailored models/tools that can adeptly handle the complexities of natural language use. Enhancing automated valence analysis is not only pivotal for advancing computational methodologies but also holds significant promise for psychological research with ecologically valid insights into human daily experiences. We advocate for increased efforts in creating comprehensive datasets & finetuning LLMs for low-resource languages like Flemish, aiming to bridge the gap between computational linguistics & emotion research.