ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Programming

TL;DR

ProgramAlly enables blind users to create custom visual access programs using multi-modal end-user programming approaches, enhancing personalization and addressing unique accessibility needs beyond existing solutions.

Contribution

This work introduces a novel system combining block programming, natural language, and programming by example for blind users to customize visual access tools.

Findings

Participants preferred different programming modalities per task.

Users envisioned addressing unique accessibility challenges.

The system supports on-device and cloud-based program generation.

Abstract

Existing visual assistive technologies are built for simple and common use cases, and have few avenues for blind people to customize their functionalities. Drawing from prior work on DIY assistive technology, this paper investigates end-user programming as a means for users to create and customize visual access programs to meet their unique needs. We introduce ProgramAlly, a system for creating custom filters for visual information, e.g., 'find NUMBER on BUS', leveraging three end-user programming approaches: block programming, natural language, and programming by example. To implement ProgramAlly, we designed a representation of visual filtering tasks based on scenarios encountered by blind people, and integrated a set of on-device and cloud models for generating and running these programs. In user studies with 12 blind adults, we found that participants preferred different programming modalities depending on the task, and envisioned using visual access programs to address unique accessibility challenges that are otherwise difficult with existing applications. Through ProgramAlly, we present an exploration of how blind end-users can create visual access programs to customize and control their experiences.