SIL: Symbiotic Interactive Learning for Language-Conditioned Human-Agent Co-Adaptation

TL;DR

This paper introduces SIL, a bidirectional co-adaptive framework for human-agent interaction that leverages foundation models and memory architectures to improve task understanding, communication, and stability in embodied tasks.

Contribution

SIL is a novel symbiotic interactive learning framework enabling reciprocal adaptation between humans and agents in shared latent spaces, enhancing multi-turn interaction capabilities.

Findings

Achieved 90.4% task completion rate in experiments.

Improved belief alignment score to 0.83, about 20% better than ablations.

Demonstrated effectiveness in simulated and real-world tasks.

Abstract

Today's autonomous agents, largely driven by foundation models (FMs), can understand natural language instructions and solve long-horizon tasks with human-like reasoning. However, current human-robot interaction largely follows a one-way master-apprentice technique where the agent passively executes commands without reciprocal learning. This neglects the co-adaptive, multi-turn nature of everyday human interactions. We introduce symbiotic interactive learning (SIL), a bidirectional co-adaptation framework in a shared latent task space, where human and agent maintain joint belief states that evolve with interaction history. This enables proactive clarification, adaptive suggestions, and shared plan refinement. SIL leverages FMs for spatial perception and reasoning, together with a triplet-loss-trained neural encoder that grounds FMs' outputs into task-specific latent representations. To support long-term stability as tasks evolve, SIL uses episodic and semantic memory architectures, regularised via elastic weight consolidation to mitigate catastrophic forgetting. We evaluate SIL on simulated and real-world embodied tasks, including instruction following, information retrieval, query-oriented reasoning, and interactive dialogue, achieving a $90.4\%$ task completion rate and a belief alignment score of $\rho \approx 0.83$, an absolute improvement of about $20$ percentage points over the best ablations. Demos and resources: https://linusnep.github.io/SIL/.