Video Games as a Corpus: Sentiment Analysis using Fallout New Vegas Dialog

TL;DR

This paper introduces a multilingual sentiment analysis dataset derived from Fallout New Vegas dialogues, and evaluates various transformer models, revealing challenges and insights in game dialogue sentiment classification.

Contribution

The paper presents a novel multilingual, multilabel sentiment dataset from a popular video game and benchmarks transformer models on this challenging dataset.

Findings

Multilingual BERT outperformed XLMRoBERTa in most languages.

Language-specific BERT models were slightly better than multilingual BERT.

Best accuracy achieved was 54% on Spanish data.

Abstract

We present a method for extracting a multilingual sentiment annotated dialog data set from Fallout New Vegas. The game developers have preannotated every line of dialog in the game in one of the 8 different sentiments: \textit{anger, disgust, fear, happy, neutral, pained, sad } and \textit{surprised}. The game has been translated into English, Spanish, German, French and Italian. We conduct experiments on multilingual, multilabel sentiment analysis on the extracted data set using multilingual BERT, XLMRoBERTa and language specific BERT models. In our experiments, multilingual BERT outperformed XLMRoBERTa for most of the languages, also language specific models were slightly better than multilingual BERT for most of the languages. The best overall accuracy was 54\% and it was achieved by using multilingual BERT on Spanish data. The extracted data set presents a challenging task for sentiment analysis. We have released the data, including the testing and training splits, openly on Zenodo. The data set has been shuffled for copyright reasons.