Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning

TL;DR

This paper introduces Action Shapley, a new metric for selecting high-quality training data for world models in reinforcement learning, improving efficiency and performance in data-constrained scenarios.

Contribution

The paper proposes Action Shapley as an unbiased data selection metric and a randomized algorithm to compute it efficiently, enhancing data quality in reinforcement learning.

Findings

Over 80% computational efficiency gain over traditional methods

Outperforms ad-hoc data selection policies in experiments

Effective in five real-world data-constrained case studies

Abstract

Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the algorithm demonstrates a computational efficiency improvement exceeding 80\% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.