Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM

TL;DR

This paper introduces a long-context Q-former integrated with a multimodal LLM to improve human-robot interaction by better understanding and generating robot actions based on extended video context and dialogue.

Contribution

It proposes a novel long-context Q-former that incorporates full video context and a text-conditioning approach for enhanced robot action confirmation and planning.

Findings

Long-context Q-former improves confirmation accuracy.

Enhanced action planning with integrated VideoLLaMA3.

Better utilization of full video context for HRI.

Abstract

Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot interaction (HRI) based on human-robot dialogue that relies on the robot action confirmation and action step generation using multimodal scene understanding. The state-of-the-art approach uses multimodal transformers to generate robot action steps aligned with robot action confirmation from a single clip showing a task composed of multiple micro steps. Although actions towards a long-horizon task depend on each other throughout an entire video, the current approaches mainly focus on clip-level processing and do not leverage long-context information. This paper proposes a long-context Q-former incorporating left and right context dependency in full videos. Furthermore, this paper proposes a text-conditioning approach to feed text embeddings directly into the LLM decoder to mitigate the high abstraction of the information in text by Q-former. Experiments with the YouCook2 corpus show that the accuracy of confirmation generation is a major factor in the performance of action planning. Furthermore, we demonstrate that the long-context Q-former improves the confirmation and action planning by integrating VideoLLaMA3.