SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering

TL;DR

SweeperBot is a novel system that enhances 3D model accessibility for screen reader users by combining view analysis and visual question answering, enabling exploration and comparison of models.

Contribution

This paper introduces SweeperBot, a new system that integrates view selection and VQA models to improve 3D model accessibility for visually impaired users.

Findings

Expert review with 10 BLV users showed feasibility.

Generated descriptions were validated by 30 sighted participants.

System effectively supports exploration and comparison of 3D models.

Abstract

Accessing 3D models remains challenging for Screen Reader (SR) users. While some existing 3D viewers allow creators to provide alternative text, they often lack sufficient detail about the 3D models. Grounded on a formative study, this paper introduces SweeperBot, a system that enables SR users to leverage visual question answering to explore and compare 3D models. SweeperBot answers SR users' visual questions by combining an optimal view selection technique with the strength of generative- and recognition-based foundation models. An expert review with 10 Blind and Low-Vision (BLV) users with SR experience demonstrated the feasibility of using SweeperBot to assist BLV users in exploring and comparing 3D models. The quality of the descriptions generated by SweeperBot was validated by a second survey study with 30 sighted participants.