HALyPO: Heterogeneous-Agent Lyapunov Policy Optimization for Human-Robot Collaboration

TL;DR

HALyPO introduces a Lyapunov-based method to stabilize multi-agent policy learning in human-robot collaboration, enhancing generalization and robustness by addressing heterogeneity and rationality gaps.

Contribution

The paper proposes HALyPO, a novel Lyapunov policy optimization framework that ensures stability in heterogeneous multi-agent reinforcement learning for HRC.

Findings

Improved generalization in human-robot collaboration tasks.

Enhanced robustness in open-ended interaction scenarios.

Validated effectiveness through simulations and humanoid-robot experiments.

Abstract

To improve generalization and resilience in human-robot collaboration (HRC), robots must handle the combinatorial diversity of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG) in the learning process-a variational mismatch between decentralized best-response dynamics and centralized cooperative ascent. The resulting learning problem is a general-sum differentiable game, so independent policy-gradient updates can oscillate or diverge without added structure. We propose heterogeneous-agent Lyapunov policy optimization (HALyPO), which establishes formal stability directly in the policy-parameter space by enforcing a per-step Lyapunov decrease condition on a parameter-space disagreement metric. Unlike Lyapunov-based safe RL, which targets state/trajectory constraints in constrained Markov decision processes, HALyPO uses Lyapunov certification to stabilize decentralized policy learning. HALyPO rectifies decentralized gradients via optimal quadratic projections, ensuring monotonic contraction of RG and enabling effective exploration of open-ended interaction spaces. Extensive simulations and real-world humanoid-robot experiments show that this certified stability improves generalization and robustness in collaborative corner cases.