VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting

TL;DR

VITA-E introduces a dual-model framework enabling embodied agents to see, hear, speak, and act concurrently with real-time interruption handling, advancing natural human-like multitasking in interactive scenarios.

Contribution

The paper presents a novel dual-model architecture and model-as-controller paradigm for concurrent, interruptible embodied interaction, improving responsiveness and multitasking capabilities.

Findings

High success rate in emergency stops and speech interruptions

Effective concurrent speech and action execution

Reliable handling of complex interactive scenarios

Abstract

Current Vision-Language-Action (VLA) models are often constrained by a rigid, static interaction paradigm, which lacks the ability to see, hear, speak, and act concurrently as well as handle real-time user interruptions dynamically. This hinders seamless embodied collaboration, resulting in an inflexible and unresponsive user experience. To address these limitations, we introduce VITA-E, a novel embodied interaction framework designed for both behavioral concurrency and nearly real-time interruption. The core of our approach is a dual-model architecture where two parallel VLA instances operate as an ``Active Model'' and a ``Standby Model'', allowing the embodied agent to observe its environment, listen to user speech, provide verbal responses, and execute actions, all concurrently and interruptibly, mimicking human-like multitasking capabilities. We further propose a ``model-as-controller'' paradigm, where we fine-tune the VLM to generate special tokens that serve as direct system-level commands, coupling the model's reasoning with the system's behavior. Experiments conducted on a physical humanoid platform demonstrate that VITA-E can reliably handle complex interactive scenarios. Our framework is compatible with various dual-system VLA models, achieving an extremely high success rate on emergency stops and speech interruptions while also successfully performing concurrent speech and action. This represents a significant step towards more natural and capable embodied assistants.