AXNav: Replaying Accessibility Tests from Natural Language

TL;DR

This paper introduces AXNav, a system that uses large language models and UI understanding to automate accessibility testing, generating navigable videos with flagged issues, thus aiding QA professionals and improving testing efficiency.

Contribution

The paper presents a novel approach combining LLMs and pixel-based UI understanding to automate accessibility testing and generate informative videos for QA professionals.

Findings

Participants found the tool very useful for their work.

The system performed tests similarly to manual testing.

Insights for future LLM applications in accessibility testing.

Abstract

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs, however to our knowledge no one has yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes as input a manual accessibility test (e.g., ``Search for a show in VoiceOver'') and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10 participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.