Redefining Robot Generalization Through Interactive Intelligence

TL;DR

This paper advocates for a shift from single-agent to multi-agent, interactive foundation models in robotics, emphasizing human-robot co-adaptation and proposing a neuroscience-inspired architecture for semi-autonomous systems.

Contribution

It introduces a novel, generalizable architecture for robot foundation models that incorporates multimodal sensing, teamwork, predictive modeling, and adaptive feedback inspired by neuroscience.

Findings

Proposes a four-module architecture for interactive robot models.

Highlights the importance of human-robot co-adaptation.

Broadly applicable to semi-autonomous and wearable robotic systems.

Abstract

Recent advances in large-scale machine learning have produced high-capacity foundation models capable of adapting to a broad array of downstream tasks. While such models hold great promise for robotics, the prevailing paradigm still portrays robots as single, autonomous decision-makers, performing tasks like manipulation and navigation, with limited human involvement. However, a large class of real-world robotic systems, including wearable robotics (e.g., prostheses, orthoses, exoskeletons), teleoperation, and neural interfaces, are semiautonomous, and require ongoing interactive coordination with human partners, challenging single-agent assumptions. In this position paper, we argue that robot foundation models must evolve to an interactive multi-agent perspective in order to handle the complexities of real-time human-robot co-adaptation. We propose a generalizable, neuroscience-inspired architecture encompassing four modules: (1) a multimodal sensing module informed by sensorimotor integration principles, (2) an ad-hoc teamwork model reminiscent of joint-action frameworks in cognitive science, (3) a predictive world belief model grounded in internal model theories of motor control, and (4) a memory/feedback mechanism that echoes concepts of Hebbian and reinforcement-based plasticity. Although illustrated through the lens of cyborg systems, where wearable devices and human physiology are inseparably intertwined, the proposed framework is broadly applicable to robots operating in semi-autonomous or interactive contexts. By moving beyond single-agent designs, our position emphasizes how foundation models in robotics can achieve a more robust, personalized, and anticipatory level of performance.