Forecasting in Offline Reinforcement Learning for Non-stationary Environments

TL;DR

This paper introduces FORL, a novel offline RL framework that combines diffusion-based state prediction and zero-shot time-series forecasting to handle abrupt, non-stationary environment changes, improving robustness and performance.

Contribution

The paper proposes a unified framework for non-stationary offline RL that does not assume specific non-stationarity patterns and leverages foundation models for robust forecasting.

Findings

FORL outperforms baseline methods on offline RL benchmarks with non-stationary data.

Integration of zero-shot forecasting enhances agent robustness in unpredictable environments.

Empirical results demonstrate improved adaptation to abrupt environment changes.

Abstract

Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assume stationarity or only consider synthetic perturbations at test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade performance. To overcome this challenge, we introduce Forecasting in Non-stationary Offline RL (FORL), a framework that unifies (i) conditional diffusion-based candidate state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves performance compared to competitive baselines. By integrating zero-shot forecasting with the agent's experience, we aim to bridge the gap between offline RL and the complexities of real-world, non-stationary environments.