A Modern System Recipe for Situated Embodied Human-Robot Conversation with Real-Time Multimodal LLMs and Tool-Calling

TL;DR

This paper introduces a minimal system combining real-time multimodal large language models with tool interfaces to enable robots to perform situated, embodied conversations involving active perception and attention shifts in real-time scenarios.

Contribution

The paper presents a simple, effective system recipe that integrates multimodal LLMs with tools for active perception, demonstrating its effectiveness in home-style scenarios.

Findings

High turn-level tool-decision accuracy against human annotations

Positive subjective ratings of interaction quality

Real-time multimodal LLMs facilitate practical embodied conversation

Abstract

Situated embodied conversation requires robots to interleave real-time dialogue with active perception: deciding what to look at, when to look, and what to say under tight latency constraints. We present a simple, minimal system recipe that pairs a real-time multimodal language model with a small set of tool interfaces for attention and active perception. We study six home-style scenarios that require frequent attention shifts and increasing perceptual scope. Across four system variants, we evaluate turn-level tool-decision correctness against human annotations and collect subjective ratings of interaction quality. Results indicate that real-time multimodal large language models and tool use for active perception is a promising direction for practical situated embodied conversation.