Offline Behavior Distillation

TL;DR

This paper introduces Offline Behavior Distillation (OBD), a method to efficiently synthesize expert-like data from sub-optimal offline RL data, enabling faster policy learning with theoretical guarantees and improved empirical performance.

Contribution

The paper proposes Av-PBC, an improved OBD objective with linear discount complexity, and provides theoretical analysis and extensive experiments demonstrating its effectiveness.

Findings

Av-PBC achieves superior distillation performance.

It converges faster than naive methods.

It generalizes well across architectures and optimizers.

Abstract

Massive reinforcement learning (RL) data are typically collected to train policies offline without the need for interactions, but the large data volume can cause training inefficiencies. To tackle this issue, we formulate offline behavior distillation (OBD), which synthesizes limited expert behavioral data from sub-optimal RL data, enabling rapid policy learning. We propose two naive OBD objectives, DBC and PBC, which measure distillation performance via the decision difference between policies trained on distilled data and either offline data or a near-expert policy. Due to intractable bi-level optimization, the OBD objective is difficult to minimize to small values, which deteriorates PBC by its distillation performance guarantee with quadratic discount complexity $\mathcal{O}(1/(1-\gamma)^2)$. We theoretically establish the equivalence between the policy performance and action-value weighted decision difference, and introduce action-value weighted PBC (Av-PBC) as a more effective OBD objective. By optimizing the weighted decision difference, Av-PBC achieves a superior distillation guarantee with linear discount complexity $\mathcal{O}(1/(1-\gamma))$. Extensive experiments on multiple D4RL datasets reveal that Av-PBC offers significant improvements in OBD performance, fast distillation convergence speed, and robust cross-architecture/optimizer generalization.