Game State Learning via Game Scene Augmentation

TL;DR

This paper introduces GameCLR, a novel game scene augmentation method leveraging game-engine capabilities to improve contrastive learning for accurate game state inference from pixel data.

Contribution

GameCLR is a new augmentation technique that enhances contrastive learning by synthesizing controlled game scenes, improving game state understanding from images.

Findings

GameCLR outperforms SimCLR in game state inference accuracy.

Using game-engine based augmentation improves contrastive learning effectiveness.

The approach enables direct use of screen pixels for AI research in gaming.

Abstract

Having access to accurate game state information is of utmost importance for any artificial intelligence task including game-playing, testing, player modeling, and procedural content generation. Self-Supervised Learning (SSL) techniques have shown to be capable of inferring accurate game state information from the high-dimensional pixel input of game footage into compressed latent representations. Contrastive Learning is a popular SSL paradigm where the visual understanding of the game's images comes from contrasting dissimilar and similar game states defined by simple image augmentation methods. In this study, we introduce a new game scene augmentation technique -- named GameCLR -- that takes advantage of the game-engine to define and synthesize specific, highly-controlled renderings of different game states, thereby, boosting contrastive learning performance. We test our GameCLR technique on images of the CARLA driving simulator environment and compare it against the popular SimCLR baseline SSL method. Our results suggest that GameCLR can infer the game's state information from game footage more accurately compared to the baseline. Our proposed approach allows us to conduct game artificial intelligence research by directly utilizing screen pixels as input.