Caption: Generating Informative Content Labels for Image Buttons Using Next-Screen Context

TL;DR

This paper introduces Caption, an LLM-powered tool that generates more accurate and descriptive content labels for mobile app image buttons by utilizing next-screen context, improving accessibility for screen reader users.

Contribution

Caption is the first system to incorporate next-screen context for label generation, enhancing label accuracy beyond on-screen-only methods.

Findings

Caption outperforms human annotators in label accuracy.

Caption surpasses baseline LLM methods in descriptive quality.

Preliminary results demonstrate improved accessibility support.

Abstract

We present Caption, an LLM-powered content label generation tool for visual interactive elements on mobile devices. Content labels are essential for screen readers to provide announcements for image-based elements, but are often missing or uninformative due to developer neglect. Automated captioning systems attempt to address this, but are limited to on-screen context, often resulting in inaccurate or unspecific labels. To generate more accurate and descriptive labels, Caption collects next-screen context on interactive elements by navigating to the destination screen that appears after an interaction and incorporating information from both the origin and destination screens. Preliminary results show Caption generates more accurate labels than both human annotators and an LLM baseline. We expect Caption to empower developers by providing actionable accessibility suggestions and directly support on-demand repairs by screen reader users.