A11yShape: AI-Assisted 3-D Modeling for Blind and Low-Vision Programmers

TL;DR

A11yShape is an innovative system that employs AI and integrated tools to enable blind and low-vision programmers to understand, modify, and create 3-D models independently, overcoming significant accessibility barriers.

Contribution

This paper introduces A11yShape, a novel AI-assisted platform that enhances non-visual interaction with 3-D modeling tools for BLV users, integrating multiple representations and synchronization mechanisms.

Findings

Participants successfully understood and modified 3-D models independently.

A11yShape improved accessibility and user satisfaction for BLV programmers.

The system enabled tasks previously impossible without sighted assistance.

Abstract

Building 3-D models is challenging for blind and low-vision (BLV) users due to the inherent complexity of 3-D models and the lack of support for non-visual interaction in existing tools. To address this issue, we introduce A11yShape, a novel system designed to help BLV users who possess basic programming skills understand, modify, and iterate on 3-D models. A11yShape leverages LLMs and integrates with OpenSCAD, a popular open-source editor that generates 3-D models from code. Key functionalities of A11yShape include accessible descriptions of 3-D models, version control to track changes in models and code, and a hierarchical representation of model components. Most importantly, A11yShape employs a cross-representation highlighting mechanism to synchronize semantic selections across all model representations -- code, semantic hierarchy, AI description, and 3-D rendering. We conducted a multi-session user study with four BLV programmers, where, after an initial tutorial session, participants independently completed 12 distinct models across two testing sessions, achieving results that aligned with their own satisfaction. The result demonstrates that participants were able to comprehend provided 3-D models, as well as independently create and modify 3-D models -- tasks that were previously impossible without assistance from sighted individuals.