CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognition

TL;DR

CARMA is a system that enhances human-robot group interaction by accurately recognizing and tracking actors, objects, and actions, enabling robots to understand and respond effectively in collaborative scenarios.

Contribution

The paper presents a novel approach combining vision-language models with object and action recognition to achieve consistent situational grounding in group interactions.

Findings

Reliable generation of actor-action-object triplets

Effective role distinction and multi-actor awareness

Robust instance identification in dynamic scenarios

Abstract

We introduce CARMA, a system for situational grounding in human-robot group interactions. Effective collaboration in such group settings requires situational awareness based on a consistent representation of present persons and objects coupled with an episodic abstraction of events regarding actors and manipulated objects. This calls for a clear and consistent assignment of instances, ensuring that robots correctly recognize and track actors, objects, and their interactions over time. To achieve this, CARMA uniquely identifies physical instances of such entities in the real world and organizes them into grounded triplets of actors, objects, and actions. To validate our approach, we conducted three experiments, where multiple humans and a robot interact: collaborative pouring, handovers, and sorting. These scenarios allow the assessment of the system's capabilities as to role distinction, multi-actor awareness, and consistent instance identification. Our experiments demonstrate that the system can reliably generate accurate actor-action-object triplets, providing a structured and robust foundation for applications requiring spatiotemporal reasoning and situated decision-making in collaborative settings.