Residual Reinforcement Learning for Robot Teleoperation under Stochastic Delays

TL;DR

This paper introduces a hybrid control framework combining LSTM-based state estimation with residual reinforcement learning to improve robot teleoperation stability under stochastic communication delays.

Contribution

The proposed delay-resilient RL method effectively reconstructs states and learns residual policies, outperforming existing methods in delay-affected robot teleoperation.

Findings

Significantly improves teleoperation stability under stochastic delays

Outperforms state-of-the-art baselines in experiments with Franka Panda robots

Ensures robust and smooth control despite high-variance communication delays

Abstract

Stochastic communication delays in teleoperation introduce signal discontinuities that undermine control stability and degrade control performance. Consequently, the conventional reinforcement learning (RL) methods struggle with the delayed observations due to the delay-induced observations, leading to high-frequency chattering. To address this, we propose a hybrid control framework, delay-resilient RL, integrating a state estimator utilizing Long Short-Term Memory (LSTM) with a residual RL policy, which is resilient to stochastic delays. The LSTM reconstructs smooth, continuous state estimates from delayed observations, enabling the RL agent to learn a residual torque compensation policy that balances tracking accuracy with velocity smoothness. Experimental validation on Franka Panda robots demonstrates that our approach significantly outperforms the state-of-the-art baselines, ensuring robust and stable teleoperation even under high-variance stochastic delays.