Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions

TL;DR

This paper introduces a system that presents multiple variations of MLLM-generated image descriptions to help blind and low vision users better detect unreliable information, improving safety and decision-making.

Contribution

It proposes a novel design space and prototype for surfacing description variations, supported by a user study demonstrating improved reliability detection.

Findings

Variations increased detection of unreliable claims by 4.9x

Participants preferred seeing multiple descriptions over a single one

All participants showed interest in using the system for real-world tasks

Abstract

Multimodal large language models (MLLMs) provide new opportunities for blind and low vision (BLV) people to access visual information in their daily lives. However, these models often produce errors that are difficult to detect without sight, posing safety and social risks in scenarios from medication identification to outfit selection. While BLV MLLM users use creative workarounds such as cross-checking between tools and consulting sighted individuals, these approaches are often time-consuming and impractical. We explore how systematically surfacing variations across multiple MLLM responses can support BLV users to detect unreliable information without visually inspecting the image. We contribute a design space for eliciting and presenting variations in MLLM descriptions, a prototype system implementing three variation presentation styles, and findings from a user study with 15 BLV participants. Our results demonstrate that presenting variations significantly increases users' ability to identify unreliable claims (by 4.9x using our approach compared to single descriptions) and significantly decreases perceived reliability of MLLM responses. 14 of 15 participants preferred seeing variations of MLLM responses over a single description, and all expressed interest in using our system for tasks from understanding a tornado's path to posting an image on social media.