Conditioning LLMs with Emotion in Neural Machine Translation

TL;DR

This paper introduces a novel machine translation approach that incorporates emotion signals from speech recognition models into large language models, significantly improving translation quality.

Contribution

It presents a new pipeline that integrates emotion information into LLM prompts for enhanced neural machine translation performance.

Findings

Emotion integration, especially arousal, improves translation quality

Fine-tuning LLMs on Libri-trans dataset is effective

Emotion-augmented prompts outperform baseline models

Abstract

Large Language Models (LLMs) have shown remarkable performance in Natural Language Processing tasks, including Machine Translation (MT). In this work, we propose a novel MT pipeline that integrates emotion information extracted from a Speech Emotion Recognition (SER) model into LLMs to enhance translation quality. We first fine-tune five existing LLMs on the Libri-trans dataset and select the most performant model. Subsequently, we augment LLM prompts with different dimensional emotions and train the selected LLM under these different configurations. Our experiments reveal that integrating emotion information, especially arousal, into LLM prompts leads to notable improvements in translation quality.