SOMBRL: Scalable and Optimistic Model-Based RL

TL;DR

SOMBRL introduces a scalable, optimistic model-based reinforcement learning method that effectively balances exploration and exploitation by incorporating uncertainty, demonstrating strong theoretical guarantees and superior empirical performance in various environments.

Contribution

The paper presents SOMBRL, a novel scalable and uncertainty-aware MBRL algorithm that guarantees sublinear regret and outperforms existing methods in simulation and real-world tasks.

Findings

SOMBRL achieves sublinear regret in nonlinear dynamics settings.

It outperforms state-of-the-art methods in both simulated and real-world environments.

The approach demonstrates strong empirical results across diverse tasks.

Abstract

We address the challenge of efficient exploration in model-based reinforcement learning (MBRL), where the system dynamics are unknown and the RL agent must learn directly from online interactions. We propose Scalable and Optimistic MBRL (SOMBRL), an approach based on the principle of optimism in the face of uncertainty. SOMBRL learns an uncertainty-aware dynamics model and greedily maximizes a weighted sum of the extrinsic reward and the agent's epistemic uncertainty. SOMBRL is compatible with any policy optimizers or planners, and under common regularity assumptions on the system, we show that SOMBRL has sublinear regret for nonlinear dynamics in the (i) finite-horizon, (ii) discounted infinite-horizon, and (iii) non-episodic settings. Additionally, SOMBRL offers a flexible and scalable solution for principled exploration. We evaluate SOMBRL on state-based and visual-control environments, where it displays strong performance across all tasks and baselines. We also evaluate SOMBRL on a dynamic RC car hardware and show SOMBRL outperforms the state-of-the-art, illustrating the benefits of principled exploration for MBRL.