Offline Behavioral Data Selection

TL;DR

This paper introduces Stepwise Dual Ranking (SDR), a method for selecting informative subsets from large offline behavioral datasets to improve policy learning efficiency and performance.

Contribution

The paper proposes SDR, a novel data selection technique that combines stepwise clipping and dual ranking to enhance offline behavioral data utility.

Findings

SDR improves policy performance with smaller data subsets.

SDR reduces computational costs in offline policy learning.

Extensive experiments validate SDR's effectiveness on D4RL benchmarks.

Abstract

Behavioral cloning is a widely adopted approach for offline policy learning from expert demonstrations. However, the large scale of offline behavioral datasets often results in computationally intensive training when used in downstream tasks. In this paper, we uncover the striking data saturation in offline behavioral data: policy performance rapidly saturates when trained on a small fraction of the dataset. We attribute this effect to the weak alignment between policy performance and test loss, revealing substantial room for improvement through data selection. To this end, we propose a simple yet effective method, Stepwise Dual Ranking (SDR), which extracts a compact yet informative subset from large-scale offline behavioral datasets. SDR is build on two key principles: (1) stepwise clip, which prioritizes early-stage data; and (2) dual ranking, which selects samples with both high action-value rank and low state-density rank. Extensive experiments and ablation studies on D4RL benchmarks demonstrate that SDR significantly enhances data selection for offline behavioral data.