Feasibility of Interactive 3D Map for Remote Sighted Assistance

TL;DR

This paper demonstrates that using interactive 3D maps with localization in remote sighted assistance significantly improves agent response times and reduces mental workload, showing promise for aiding users with vision impairments.

Contribution

The paper introduces a high-fidelity prototype of RSA utilizing interactive 3D maps with localization, addressing key challenges faced by sighted agents.

Findings

Agents provided faster assistance with 3D maps

Mental workload for agents was reduced

Feasibility of 3D maps in RSA was confirmed

Abstract

Remote sighted assistance (RSA) has emerged as a conversational assistive technology, where remote sighted workers, i.e., agents, provide real-time assistance to users with vision impairments via video-chat-like communication. Researchers found that agents' lack of environmental knowledge, the difficulty of orienting users in their surroundings, and the inability to estimate distances from users' camera feeds are key challenges to sighted agents. To address these challenges, researchers have suggested assisting agents with computer vision technologies, especially 3D reconstruction. This paper presents a high-fidelity prototype of such an RSA, where agents use interactive 3D maps with localization capability. We conducted a walkthrough study with thirteen agents and one user with simulated vision impairment using this prototype. The study revealed that, compared to baseline RSA, the agents were significantly faster in providing navigational assistance to users, and their mental workload was significantly reduced -- all indicate the feasibility and prospect of 3D maps in RSA.