AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant
Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei, Gao, Mike Zheng Shou

TL;DR
This paper introduces a new task called Affordance-centric Question-driven Task Completion for egocentric AI assistants, along with a dataset and a model, to improve step-by-step real-world assistance from instructional videos.
Contribution
The paper defines a novel task, creates the AssistQ dataset with 531 QA samples from instructional videos, and proposes the Q2A model to advance egocentric AI assistant capabilities.
Findings
Q2A model outperforms VQA baselines
AssistQ dataset enables new research directions
Significant room for improvement remains
Abstract
A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as "how can I run the microwave for 1 minute?". However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view. To support the task, we constructed AssistQ, a new dataset comprising 531 question-answer samples from 100 newly filmed instructional videos. We also developed a novel Question-to-Actions (Q2A) model to address the AQTC task and validate it on the AssistQ dataset. The results show that our model significantly outperforms several VQA-related baselines while still having large room for improvement. We expect our task and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · AI in Service Interactions · Artificial Intelligence in Healthcare and Education
