Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning

TL;DR

This paper introduces the COWM layer, a clustering-based weight orthogonalization technique that stabilizes deep reinforcement learning, improves sample efficiency, and outperforms existing methods across multiple benchmarks.

Contribution

The paper proposes the COWM layer, a novel method integrating clustering and projection to mitigate non-stationarity in RL, enhancing stability and learning speed.

Findings

COWM outperforms state-of-the-art methods by 9% and 12.6% on benchmarks.

COWM reduces gradient interference and stabilizes learning.

The approach is robust and general across various RL algorithms.

Abstract

Reinforcement learning (RL) has made significant advancements, achieving superhuman performance in various tasks. However, RL agents often operate under the assumption of environmental stationarity, which poses a great challenge to learning efficiency since many environments are inherently non-stationary. This non-stationarity results in the requirement of millions of iterations, leading to low sample efficiency. To address this issue, we introduce the Clustering Orthogonal Weight Modified (COWM) layer, which can be integrated into the policy network of any RL algorithm and mitigate non-stationarity effectively. The COWM layer stabilizes the learning process by employing clustering techniques and a projection matrix. Our approach not only improves learning speed but also reduces gradient interference, thereby enhancing the overall learning efficiency. Empirically, the COWM outperforms state-of-the-art methods and achieves improvements of 9% and 12.6% in vision based and state-based DMControl benchmark. It also shows robustness and generality across various algorithms and tasks.