"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models

TL;DR

This study evaluates how image quality issues like blur and misframing impact the accuracy of vision-language models in generating product captions for blind and low-vision users, highlighting significant accuracy drops with poor quality images.

Contribution

It provides an empirical dataset and analysis of image quality effects on VLM caption accuracy specifically for blind and low-vision users, emphasizing the need for inclusive evaluation.

Findings

Caption accuracy drops from 98% to 75% with image quality issues.

Accuracy worsens as multiple image quality issues compound.

Recommendations are provided for more inclusive model evaluation.

Abstract

Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal care items, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues--such as blur, misframing, and rotation--affect the accuracy of VLM-generated captions and whether the resulting captions meet BLV people's information needs. Based on a survey of 86 BLV participants, we develop an annotated dataset of 1,859 product images from BLV people to systematically evaluate how image quality issues affect VLM-generated captions. While the best VLM achieves 98% accuracy on images with no quality issues, accuracy drops to 75% overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.