Event-Driven Proactive Assistive Manipulation with Grounded Vision-Language Planning

TL;DR

This paper introduces an event-driven framework for proactive robot assistance in collaborative tasks, enabling robots to infer when to intervene based on observed workspace state changes rather than user instructions.

Contribution

It presents a novel event-driven approach that uses workspace state transitions to guide proactive assistance, improving robot responsiveness in collaborative manipulation tasks.

Findings

Enhanced proactive assistance in real-world tasks

Improved intervention accuracy based on state changes

Effective handling of solvable and unsolvable scenes

Abstract

Assistance in collaborative manipulation is often initiated by user instructions, making high-level reasoning request-driven. In fluent human teamwork, however, partners often infer the next helpful step from the observed outcome of an action rather than waiting for instructions. Motivated by this, we introduce a shift from request-driven assistance to event-driven proactive assistance, where robot actions are initiated by workspace state transitions induced by human--object interactions rather than user-provided task instructions. To this end, we propose an event-driven framework that tracks interaction progress with an event monitor and, upon event completion, extracts stabilized pre/post snapshots that characterize the resulting state transition. Given the stabilized snapshots, the planner analyzes the implied state transition to infer a task-level goal and decide whether to intervene; if so, it generates a sequence of assistive actions. To make outputs executable and verifiable, we restrict actions to a set of action primitives and reference objects via integer IDs. We evaluate the framework on a real tabletop number-block collaboration task, demonstrating that explicit pre/post state-change evidence improves proactive completion on solvable scenes and appropriate waiting on unsolvable ones.