Beyond Semantics: Measuring Fine-Grained Emotion Preservation in Small Language Model-Based Machine Translation

TL;DR

This paper evaluates how well small language models preserve fine-grained emotions in machine translation, focusing on emotion retention, prompting techniques, and classification methods across multiple European languages.

Contribution

It introduces an evaluation framework for emotional fidelity in MT using SLMs, emotion-aware prompting, and ModernBERT, highlighting challenges and potential improvements.

Findings

SLMs show limited inherent emotional preservation.

Emotion-aware prompting improves emotion retention.

ModernBERT enhances emotion classification accuracy.

Abstract

Preserving affective nuance remains a challenge in Machine Translation (MT), where semantic equivalence often takes precedence over emotional fidelity. This paper evaluates the performance of three state-of-the-art Small Language Models (SLMs) -- EuroLLM, Aya Expanse, and Gemma -- in maintaining fine-grained emotions during backtranslation. Using the GoEmotions dataset, which comprises Reddit comments across 28 distinct categories, we assess emotional preservation across five European languages: German, French, Spanish, Italian, and Polish. Specifically, we investigate (i) the inherent capability of these SLMs to retain emotional sentiment, (ii) the efficacy of emotion-aware prompting in improving preservation, and (iii) the performance of ModernBERT as a contemporary alternative to BERT for emotion classification in MT evaluation.