ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response

TL;DR

This paper introduces ProAct-75, a comprehensive benchmark for training and evaluating proactive agents across various domains, and proposes ProAct-Helper, a multimodal framework that leverages task graphs and heuristic search for complex, structure-aware decision-making.

Contribution

The paper presents a new benchmark dataset with detailed annotations and task graphs, and a multimodal baseline model that improves proactive decision-making through structural reasoning and parallel action execution.

Findings

ProAct-Helper outperforms strong models in trigger detection (6.21% mF1 improvement)

It reduces decision steps by 0.25 on average in online tasks

Increases parallel action rate by 15.58%

Abstract

While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human's next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58%.