Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search

TL;DR

This paper introduces HOTD-Bench and CMAST, a new framework and benchmark for enabling large multimodal models to discover human-centric tasks in open-future scenarios, improving task discovery and reducing human effort.

Contribution

It formalizes the HOTD problem, creates HOTD-Bench for evaluation, and proposes CMAST, a scalable multi-agent search framework that enhances task discovery in open-future contexts.

Findings

CMAST outperforms existing models on HOTD-Bench.

HOTD-Bench provides a large-scale real-world video dataset.

CMAST integrates with LMMs to improve task discovery.

Abstract

Recent progress in robotics and embodied AI is largely driven by Large Multimodal Models (LMMs). However, a key challenge remains underexplored: how can we advance LMMs to discover tasks that assist humans in open-future scenarios, where human intentions are highly concurrent and dynamic. In this work, we formalize the problem of Human-centric Open-future Task Discovery (HOTD), focusing particularly on identifying tasks that reduce human effort across plausible futures. To facilitate this study, we propose HOTD-Bench, which features over 2K real-world videos, a semi-automated annotation pipeline, and a simulation-based protocol tailored for open-set future evaluation. Additionally, we propose the Collaborative Multi-Agent Search Tree (CMAST) framework, which decomposes complex reasoning through a multi-agent system and structures the reasoning process through a scalable search tree module. In our experiments, CMAST achieves the best performance on the HOTD-Bench, significantly surpassing existing LMMs. It also integrates well with existing LMMs, consistently improving performance.