# Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams

## TL;DR

This paper introduces a new dataset of League of Legends livestreams annotated for emotion and game context, and proposes a tensor decomposition-based multimodal fusion method for joint prediction of affect and game state.

## Contribution

It presents the first annotated dataset for affect and game context in livestreams and introduces a novel tensor-based fusion technique for multimodal data analysis.

## Key findings

- Tensor decomposition improves multimodal fusion accuracy
- Joint prediction of affect and game context outperforms baseline methods
- New dataset enables advanced research in livestream analysis

## Abstract

Video game streaming provides the viewer with a rich set of audio-visual data, conveying information both with regards to the game itself, through game footage and audio, as well as the streamer's emotional state and behaviour via webcam footage and audio. Analysing player behaviour and discovering correlations with game context is crucial for modelling and understanding important aspects of livestreams, but comes with a significant set of challenges - such as fusing multimodal data captured by different sensors in uncontrolled ('in-the-wild') conditions. Firstly, we present, to our knowledge, the first data set of League of Legends livestreams, annotated for both streamer affect and game context. Secondly, we propose a method that exploits tensor decompositions for high-order fusion of multimodal representations. The proposed method is evaluated on the problem of jointly predicting game context and player affect, compared with a set of baseline fusion approaches such as late and early fusion.