Nightmare Dreamer: Dreaming About Unsafe States And Planning Ahead

TL;DR

Nightmare Dreamer is a model-based Safe Reinforcement Learning algorithm that predicts safety violations using a learned world model, significantly reducing safety violations while maintaining high reward performance in robotics tasks.

Contribution

It introduces Nightmare Dreamer, a novel model-based Safe RL method that effectively predicts unsafe states and plans safely, outperforming existing model-free approaches.

Findings

Nearly zero safety violations achieved

20x efficiency improvement over baselines

Effective with only image observations

Abstract

Reinforcement Learning (RL) has shown remarkable success in real-world applications, particularly in robotics control. However, RL adoption remains limited due to insufficient safety guarantees. We introduce Nightmare Dreamer, a model-based Safe RL algorithm that addresses safety concerns by leveraging a learned world model to predict potential safety violations and plan actions accordingly. Nightmare Dreamer achieves nearly zero safety violations while maximizing rewards. Nightmare Dreamer outperforms model-free baselines on Safety Gymnasium tasks using only image observations, achieving nearly a 20x improvement in efficiency.