Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset

TL;DR

This paper introduces GAME-MUG, a comprehensive multimodal dataset and baseline model for understanding esports game situations and generating audience-engaged commentary, incorporating text, audio, and event data from live streams.

Contribution

It presents a new multimodal dataset for esports understanding and a joint multimodal dual learning model for improved commentary generation.

Findings

The dataset covers game situation, audience talk, and emotions from 2020-2022 streams.

The baseline model effectively integrates multimodal data for game understanding.

Results show enhanced commentary quality with multimodal and joint learning approaches.

Abstract

The dynamic nature of esports makes the situation relatively complicated for average viewers. Esports broadcasting involves game expert casters, but the caster-dependent game commentary is not enough to fully understand the game situation. It will be richer by including diverse multimodal esports information, including audiences' talks/emotions, game audio, and game match event information. This paper introduces GAME-MUG, a new multimodal game situation understanding and audience-engaged commentary generation dataset and its strong baseline. Our dataset is collected from 2020-2022 LOL game live streams from YouTube and Twitch, and includes multimodal esports game information, including text, audio, and time-series event logs, for detecting the game situation. In addition, we also propose a new audience conversation augmented commentary dataset by covering the game situation and audience conversation understanding, and introducing a robust joint multimodal dual learning model as a baseline. We examine the model's game situation/event understanding ability and commentary generation capability to show the effectiveness of the multimodal aspects coverage and the joint integration learning approach.