Vid2Coach: Transforming How-To Videos into Task Assistants
Mina Huh, Zihui Xue, Ujjaini Das, Kumar Ashutosh, Kristen Grauman, Amy Pavel

TL;DR
Vid2Coach is an AI-powered system that transforms instructional videos into accessible, real-time task assistance for blind and low vision users, improving task accuracy and user confidence.
Contribution
The paper introduces Vid2Coach, a novel system that converts how-to videos into wearable, accessible guides with real-time feedback for BLV individuals, integrating visual and non-visual cues.
Findings
BLV users made 58.5% fewer errors with Vid2Coach.
Participants expressed high willingness to adopt Vid2Coach in daily life.
The system effectively combines video augmentation with real-time user monitoring.
Abstract
People use videos to learn new recipes, exercises, and crafts. Such videos remain difficult for blind and low vision (BLV) people to follow as they rely on visual comparison. Our observations of visual rehabilitation therapists (VRTs) guiding BLV people to follow how-to videos revealed that VRTs provide both proactive and responsive support including detailed descriptions, non-visual workarounds, and progress feedback. We propose Vid2Coach, a system that transforms how-to videos into wearable camera-based assistants that provide accessible instructions and mixed-initiative feedback. From the video, Vid2Coach generates accessible instructions by augmenting narrated instructions with demonstration details and completion criteria for each step. It then uses retrieval-augmented-generation to extract relevant non-visual workarounds from BLV-specific resources. Vid2Coach then monitors user…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Multimodal Machine Learning Applications · Recommender Systems and Techniques
