'What did the Robot do in my Absence?' Video Foundation Models to Enhance Intermittent Supervision

TL;DR

This paper explores using Video Foundation Models to generate summaries of robot vision data, improving human operators' ability to monitor robot actions during long, unsupervised periods through multi-modal, query-driven summaries.

Contribution

It introduces a novel zero-shot framework utilizing ViFMs for multi-modal robot data summaries, enhancing intermittent supervision in human-robot interaction.

Findings

Query-driven summaries improve retrieval accuracy

Storyboards are most effective for object queries

Summaries increase task duration but aid in data retrieval

Abstract

This paper investigates the application of Video Foundation Models (ViFMs) for generating robot data summaries to enhance intermittent human supervision of robot teams. We propose a novel framework that produces both generic and query-driven summaries of long-duration robot vision data in three modalities: storyboards, short videos, and text. Through a user study involving 30 participants, we evaluate the efficacy of these summary methods in allowing operators to accurately retrieve the observations and actions that occurred while the robot was operating without supervision over an extended duration (40 min). Our findings reveal that query-driven summaries significantly improve retrieval accuracy compared to generic summaries or raw data, albeit with increased task duration. Storyboards are found to be the most effective presentation modality, especially for object-related queries. This work represents, to our knowledge, the first zero-shot application of ViFMs for generating multi-modal robot-to-human communication in intermittent supervision contexts, demonstrating both the promise and limitations of these models in human-robot interaction (HRI) scenarios.