Uncertainty-Aware Decision Transformer for Stochastic Driving Environments

TL;DR

This paper presents UNREST, a novel uncertainty-aware decision transformer for stochastic driving environments, improving offline RL by estimating uncertainties and using truncated returns for more reliable planning.

Contribution

Introduces UNREST, which estimates uncertainties via mutual information and employs truncated returns, enhancing offline RL in stochastic driving scenarios without complex models.

Findings

UNREST outperforms existing methods in various driving scenarios.

Uncertainty estimation improves planning safety and reliability.

Truncated returns reduce the impact of environment stochasticity.

Abstract

Offline Reinforcement Learning (RL) enables policy learning without active interactions, making it especially appealing for self-driving tasks. Recent successes of Transformers inspire casting offline RL as sequence modeling, which, however, fails in stochastic environments with incorrect assumptions that identical actions can consistently achieve the same goal. In this paper, we introduce an UNcertainty-awaRE deciSion Transformer (UNREST) for planning in stochastic driving environments without introducing additional transition or complex generative models. Specifically, UNREST estimates uncertainties by conditional mutual information between transitions and returns. Discovering 'uncertainty accumulation' and 'temporal locality' properties of driving environments, we replace the global returns in decision transformers with truncated returns less affected by environments to learn from actual outcomes of actions rather than environment transitions. We also dynamically evaluate uncertainty at inference for cautious planning. Extensive experiments demonstrate UNREST's superior performance in various driving scenarios and the power of our uncertainty estimation strategy.