Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users

TL;DR

This study explores how blind users customize interactions with a conversational visual question answering system, revealing the importance of control and prompting techniques to improve accessibility and user experience.

Contribution

It introduces a detailed analysis of user-driven customization techniques in VQA for blind users and provides a new dataset to inform accessible interaction design.

Findings

Participants often used lengthy multi-turn interactions.

Prompt engineering helped users overcome system limitations.

The study offers insights for designing more accessible VQA systems.

Abstract

Prompting and steering techniques are well established in general-purpose generative AI, yet assistive visual question answering (VQA) tools for blind users still follow rigid interaction patterns with limited opportunities for customization. User control can be helpful when system responses are misaligned with their goals and contexts, a gap that becomes especially consequential for blind users that may rely on these systems for access. We invite 11 blind users to customize their interactions with a real-world conversational VQA system. Drawing on 418 interactions, reflections, and post-study interviews, we analyze prompting-based techniques participants adopted, including those introduced in the study and those developed independently in real-world settings. VQA interactions were often lengthy: participants averaged 3 turns, sometimes up to 21, with input text typically tenfold shorter than the responses they heard. Built on state-of-the-art LLMs, the system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance. We discuss how customization techniques such as prompt engineering can help participants work around these limitations. Alongside a new publicly available dataset, we offer insights for interaction design at both query and system levels.