GamerAstra: Supporting 2D Non-Twitch Video Games for Blind and Low-Vision Players through a Multi-Agent Framework

TL;DR

GamerAstra is a multi-agent framework that improves accessibility for blind and low-vision players in 2D non-twitch video games by integrating vision-language models and customizable assistance.

Contribution

It introduces a novel multi-agent human-AI collaboration framework that enables interaction with inaccessible games without native accessibility features.

Findings

Enhances game accessibility for BLV players.

Improves interface navigation with multiple input modalities.

User studies show increased playability and immersion.

Abstract

Blind and low-vision (BLV) players face critical challenges in engaging with video games due to the inaccessibility of visual elements, difficulties navigating interfaces, and limitations in performing interaction. Meanwhile, the development of specialized accessibility features typically requires substantial programming effort and is often implemented on a game-by-game basis. To address these challenges, we introduce GamerAstra, a multi-agent human-AI collaboration framework that leverages a multi-agent design to facilitate access to 2D non-twitch video games for BLV players. It integrates vision-language models and computer vision techniques, enabling interaction with games lacking native accessibility support. The framework also incorporates custom assistance granularities to support varying degrees of visual impairment and enhances interface navigation through multiple input modalities. Technical evaluations and user studies indicate that GamerAstra effectively enhances playability and provides a more immersive gaming experience for BLV players. These findings also underscore potential avenues for advancing intelligent accessibility frameworks in the gaming domain.