Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization

TL;DR

This paper introduces DT-CORL, a belief-based offline RL framework that uses transformer models to generate delay-robust actions, effectively bridging the gap between simulation and real-world delayed environments.

Contribution

The paper presents a novel offline RL method that leverages transformers to handle delays during deployment without requiring delayed observations during training.

Findings

DT-CORL outperforms baseline methods in delay-robustness.

It improves sample efficiency compared to naive history augmentation.

It narrows the sim-to-real latency gap in benchmarks.

Abstract

Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than na\"ive history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.