StateScribe: Towards Accessible Change Awareness Across Real-World Revisits

TL;DR

StateScribe is a system designed to help blind and low-vision individuals understand real-world changes over time by tracking and describing modifications across multiple revisits, enhancing safety and awareness.

Contribution

It introduces a dual-layer memory architecture for scalable, structured change tracking and provides accurate, low-latency change descriptions in real-world environments.

Findings

Achieved 83.1% F1-score in change detection across 11 revisits.

Maintains low-latency (<1.54s) and memory usage (<54MB) over 110 revisits.

User study shows improved change awareness for BLV users in real-world locations.

Abstract

Real-world environments evolve continuously, yet blind and low-vision (BLV) individuals often have limited access to understanding how they change over time. Unexpected or relocated objects, layout modifications, and content updates (e.g., price changes) can introduce safety risks and cognitive burden. While existing visual assistive technologies can describe immediate surroundings, they operate as one-off interactions and lack mechanisms to surface meaningful changes across revisits. Informed by a survey of 33 BLV individuals, we develop StateScribe, a system that supports accessible awareness of real-world changes across revisits. StateScribe employs a dual-layer memory architecture that integrates episodic scene memory and object-centric temporal memory to enable scalable and structured change tracking. It provides both live descriptions of the current scene, and descriptions of what has changed, when and where it occurred across revisits, such as "The shop on your right has a "CLOSED" sign; it was open at this time last week.'' Our evaluation shows that StateScribe maintains high accuracy (F1-score=83.1%) across 11 revisits, while remaining low-latency (mean<1.54s) and memory-efficient (<54MB) across 110 revisits. A user study with nine BLV participants demonstrates that StateScribe improves change awareness across revisits in three real-world locations. Finally, we discuss implications for long-term AI-assisted companions that support broader change observation using multimodal sensing, extend beyond changes to other memory capabilities, and adapt to individual users, intents, and contexts.