Visual Encoders for Data-Efficient Imitation Learning in Modern Video Games

TL;DR

This paper systematically evaluates the effectiveness of pre-trained visual encoders versus end-to-end training for imitation learning in modern video games, demonstrating that pre-trained encoders can improve decision-making and reduce training costs.

Contribution

It provides a comparative analysis of pre-trained versus task-specific visual encoders for imitation learning in modern video games, highlighting the benefits of using pre-trained models.

Findings

Pre-trained encoders like DINOv2 improve decision-making performance.

End-to-end training is effective with low-resolution images and minimal demonstrations.

Pre-trained encoders significantly reduce training costs and complexity.

Abstract

Video games have served as useful benchmarks for the decision-making community, but going beyond Atari games towards modern games has been prohibitively expensive for the vast majority of the research community. Prior work in modern video games typically relied on game-specific integration to obtain game features and enable online training, or on existing large datasets. An alternative approach is to train agents using imitation learning to play video games purely from images. However, this setting poses a fundamental question: which visual encoders obtain representations that retain information critical for decision making? To answer this question, we conduct a systematic study of imitation learning with publicly available pre-trained visual encoders compared to the typical task-specific end-to-end training approach in Minecraft, Counter-Strike: Global Offensive, and Minecraft Dungeons. Our results show that end-to-end training can be effective with comparably low-resolution images and only minutes of demonstrations, but significant improvements can be gained by utilising pre-trained encoders such as DINOv2 depending on the game. In addition to enabling effective decision making, we show that pre-trained encoders can make decision-making research in video games more accessible by significantly reducing the cost of training.