Information-Theoretic Detection of Bimanual Interactions for Dual-Arm Robot Plan Generation

TL;DR

This paper introduces a novel information-theoretic approach to detect bimanual interactions from a single RGB video, enabling automatic generation of dual-arm robot plans with improved coordination understanding.

Contribution

It presents a one-shot method using Shannon's information theory and scene graph analysis to generate modular behavior trees for bimanual robot tasks from minimal demonstration data.

Findings

Effective detection of hand coordination policies

Significant improvement over existing methods in plan generation

Validated with multiple subject demonstrations and open-source datasets

Abstract

Programming by demonstration is a strategy to simplify the robot programming process for non-experts via human demonstrations. However, its adoption for bimanual tasks is an underexplored problem due to the complexity of hand coordination, which also hinders data recording. This paper presents a novel one-shot method for processing a single RGB video of a bimanual task demonstration to generate an execution plan for a dual-arm robotic system. To detect hand coordination policies, we apply Shannon's information theory to analyze the information flow between scene elements and leverage scene graph properties. The generated plan is a modular behavior tree that assumes different structures based on the desired arms coordination. We validated the effectiveness of this framework through multiple subject video demonstrations, which we collected and made open-source, and exploiting data from an external, publicly available dataset. Comparisons with existing methods revealed significant improvements in generating a centralized execution plan for coordinating two-arm systems.