LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

TL;DR

The paper introduces LEMMA, a comprehensive multi-view dataset designed to advance understanding of goal-directed, multi-task, multi-agent human activities, with benchmarks for compositional action recognition and anticipation.

Contribution

It provides a novel dataset with detailed annotations and benchmarks addressing missing dimensions in activity understanding, such as multi-agent collaboration and task scheduling.

Findings

Baseline models reveal challenges in compositional action understanding.

The dataset enables evaluation of multi-agent activity recognition.

Benchmarks highlight the need for improved temporal reasoning.

Abstract

Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence. However, a few imperative components of daily human activities are largely missed in prior literature, including the goal-directed actions, concurrent multi-tasks, and collaborations among multi-agents. We introduce the LEMMA dataset to provide a single home to address these missing dimensions with meticulously designed settings, wherein the number of tasks and agents varies to highlight different learning objectives. We densely annotate the atomic-actions with human-object interactions to provide ground-truths of the compositionality, scheduling, and assignment of daily activities. We further devise challenging compositional action recognition and action/task anticipation benchmarks with baseline models to measure the capability of compositional action understanding and temporal reasoning. We hope this effort would drive the machine vision community to examine goal-directed human activities and further study the task scheduling and assignment in the real world.