Grounding Task Assistance with Multimodal Cues from a Single Demonstration

TL;DR

This paper introduces MICA, a multimodal framework that enhances task assistance by integrating eye gaze and speech cues to better understand fine-grained user intent and context from demonstrations.

Contribution

MICA is the first framework to combine eye gaze and speech cues for improved contextual understanding in task assistance, surpassing frame-based methods.

Findings

Gaze cues alone achieve 93% of speech performance.

Multimodal cues significantly improve response accuracy.

Task type influences the effectiveness of implicit vs. explicit cues.

Abstract

A person's demonstration often serves as a key reference for others learning the same task. However, RGB video, the dominant medium for representing these demonstrations, often fails to capture fine-grained contextual cues such as intent, safety-critical environmental factors, and subtle preferences embedded in human behavior. This sensory gap fundamentally limits the ability of Vision Language Models (VLMs) to reason about why actions occur and how they should adapt to individual users. To address this, we introduce MICA (Multimodal Interactive Contextualized Assistance), a framework that improves conversational agents for task assistance by integrating eye gaze and speech cues. MICA segments demonstrations into meaningful sub-tasks and extracts keyframes and captions that capture fine-grained intent and user-specific cues, enabling richer contextual grounding for visual question answering. Evaluations on questions derived from real-time chat-assisted task replication show that multimodal cues significantly improve response quality over frame-based retrieval. Notably, gaze cues alone achieves 93% of speech performance, and their combination yields the highest accuracy. Task type determines the effectiveness of implicit (gaze) vs. explicit (speech) cues, underscoring the need for adaptable multimodal models. These results highlight the limitations of frame-based context and demonstrate the value of multimodal signals for real-world AI task assistance.