SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation

TL;DR

SimuDICE is a novel offline reinforcement learning framework that iteratively improves policies by refining synthetic experiences from a world model using distribution correction and confidence estimation, leading to efficient and robust policy optimization.

Contribution

It introduces a method combining world model updates with distribution correction via DICE to enhance offline policy learning, requiring fewer data and planning steps.

Findings

Achieves comparable performance with less data and planning.

Remains robust across different data collection policies.

Effectively balances experience distribution mismatch.

Abstract

In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experiences similar to those frequently encountered with ones that have a distribution mismatch. Our experiments show that SimuDICE achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.