Level Up Your Tutorials: VLMs for Game Tutorials Quality Assessment

TL;DR

This paper introduces an automated method using Vision-Language Models to evaluate and improve the quality of game tutorials by analyzing tutorial videos and providing feedback, reducing manual testing efforts.

Contribution

It presents a novel VLM-based approach for automated game tutorial assessment, enabling efficient detection of confusing scenes and potential errors.

Findings

VLMs can effectively analyze game tutorial frames

The approach reduces manual testing time

It improves tutorial quality assessment accuracy

Abstract

Designing effective game tutorials is crucial for a smooth learning curve for new players, especially in games with many rules and complex core mechanics. Evaluating the effectiveness of these tutorials usually requires multiple iterations with testers who have no prior knowledge of the game. Recent Vision-Language Models (VLMs) have demonstrated significant capabilities in understanding and interpreting visual content. VLMs can analyze images, provide detailed insights, and answer questions about their content. They can recognize objects, actions, and contexts in visual data, making them valuable tools for various applications, including automated game testing. In this work, we propose an automated game-testing solution to evaluate the quality of game tutorials. Our approach leverages VLMs to analyze frames from video game tutorials, answer relevant questions to simulate human perception, and provide feedback. This feedback is compared with expected results to identify confusing or problematic scenes and highlight potential errors for developers. In addition, we publish complete tutorial videos and annotated frames from different game versions used in our tests. This solution reduces the need for extensive manual testing, especially by speeding up and simplifying the initial development stages of the tutorial to improve the final game experience.