EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

TL;DR

EditScribe is a system that enables blind and low-vision users to perform and verify image edits through natural language interactions, making visual editing more accessible and controllable without visual feedback.

Contribution

This work introduces a novel natural language verification loop system that allows non-visual image editing and feedback for visually impaired users, leveraging large multimodal models.

Findings

Supported non-visual image editing and verification for blind users

Participants used diverse prompting strategies

Users found the verification feedback helpful

Abstract

Image editing is an iterative process that requires precise visual evaluation and manipulation for the output to match the editing intent. However, current image editing tools do not provide accessible interaction nor sufficient feedback for blind and low vision individuals to achieve this level of control. To address this, we developed EditScribe, a prototype system that makes image editing accessible using natural language verification loops powered by large multimodal models. Using EditScribe, the user first comprehends the image content through initial general and object descriptions, then specifies edit actions using open-ended natural language prompts. EditScribe performs the image edit, and provides four types of verification feedback for the user to verify the performed edit, including a summary of visual changes, AI judgement, and updated general and object descriptions. The user can ask follow-up questions to clarify and probe into the edits or verification feedback, before performing another edit. In a study with ten blind or low-vision users, we found that EditScribe supported participants to perform and verify image edit actions non-visually. We observed different prompting strategies from participants, and their perceptions on the various types of verification feedback. Finally, we discuss the implications of leveraging natural language verification loops to make visual authoring non-visually accessible.