AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant

TL;DR

This paper introduces a new task called Affordance-centric Question-driven Task Completion for egocentric AI assistants, along with a dataset and a model, to improve step-by-step real-world assistance from instructional videos.

Contribution

The paper defines a novel task, creates the AssistQ dataset with 531 QA samples from instructional videos, and proposes the Q2A model to advance egocentric AI assistant capabilities.

Findings

Q2A model outperforms VQA baselines

AssistQ dataset enables new research directions

Significant room for improvement remains

Abstract

A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as "how can I run the microwave for 1 minute?". However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view. To support the task, we constructed AssistQ, a new dataset comprising 531 question-answer samples from 100 newly filmed instructional videos. We also developed a novel Question-to-Actions (Q2A) model to address the AQTC task and validate it on the AssistQ dataset. The results show that our model significantly outperforms several VQA-related baselines while still having large room for improvement. We expect our task and dataset to advance Egocentric AI Assistant's development. Our project page is available at: https://showlab.github.io/assistq/.