Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

TL;DR

This paper introduces TQC, a novel method combining distributional critics, truncation, and ensembling to effectively reduce overestimation bias in continuous control, leading to significant performance improvements.

Contribution

TQC is the first approach to integrate distributional critics, truncation, and ensembling specifically for controlling overestimation bias in continuous control tasks.

Findings

TQC outperforms existing methods on all benchmark environments.

Achieves 25% improvement on the Humanoid environment.

Effectively reduces overestimation bias in off-policy learning.

Abstract

The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics. Distributional representation and truncation allow for arbitrary granular overestimation control, while ensembling provides additional score improvements. TQC outperforms the current state of the art on all environments from the continuous control benchmark suite, demonstrating 25% improvement on the most challenging Humanoid environment.