State-Action Inpainting Diffuser for Continuous Control with Delay

TL;DR

This paper introduces SAID, a novel framework combining model-based and model-free approaches to address signal delay in continuous control and RL, achieving state-of-the-art results.

Contribution

SAID formulates delay handling as a joint sequence inpainting task, bridging dynamics learning and policy optimization in RL.

Findings

SAID outperforms existing methods on delayed control benchmarks.

It demonstrates robustness in both online and offline RL settings.

The approach effectively captures environmental dynamics through sequence inpainting.

Abstract

Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.