Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge

TL;DR

This paper introduces causal exploration, a method that uses causal knowledge to improve data efficiency and reliability in task-agnostic reinforcement learning, supported by theoretical guarantees and empirical validation.

Contribution

It presents a novel causal exploration strategy that actively guides data collection and model training, enhancing sample efficiency in world model learning.

Findings

Causal exploration improves world model accuracy with less data.

Theoretical guarantees ensure convergence of the method.

Empirical results validate effectiveness on synthetic and real data.

Abstract

The effectiveness of model training heavily relies on the quality of available training resources. However, budget constraints often impose limitations on data collection efforts. To tackle this challenge, we introduce causal exploration in this paper, a strategy that leverages the underlying causal knowledge for both data collection and model training. We, in particular, focus on enhancing the sample efficiency and reliability of the world model learning within the domain of task-agnostic reinforcement learning. During the exploration phase, the agent actively selects actions expected to yield causal insights most beneficial for world model training. Concurrently, the causal knowledge is acquired and incrementally refined with the ongoing collection of data. We demonstrate that causal exploration aids in learning accurate world models using fewer data and provide theoretical guarantees for its convergence. Empirical experiments, on both synthetic data and real-world applications, further validate the benefits of causal exploration.