ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization
Tianying Ji, Yongyuan Liang, Yan Zeng, Yu Luo, Guowei Xu, Jiawei Guo,, Ruijie Zheng, Furong Huang, Fuchun Sun, Huazhe Xu

TL;DR
This paper introduces ACE, a causality-aware entropy regularization method for off-policy actor-critic algorithms that improves exploration and performance in continuous control tasks by prioritizing impactful actions and preventing gradient dormancy.
Contribution
The paper proposes a novel causality-aware entropy term and a dormancy-guided reset mechanism, enhancing exploration and sample efficiency in off-policy RL algorithms.
Findings
Achieves superior performance on 29 continuous control tasks
Effectively identifies and prioritizes impactful primitive behaviors
Demonstrates versatility and sample efficiency across domains
Abstract
The varying significance of distinct primitive behaviors during the policy learning process has been overlooked by prior model-free RL algorithms. Leveraging this insight, we explore the causal relationship between different action dimensions and rewards to evaluate the significance of various primitive behaviors during training. We introduce a causality-aware entropy term that effectively identifies and prioritizes actions with high potential impacts for efficient exploration. Furthermore, to prevent excessive focus on specific primitive behaviors, we analyze the gradient dormancy phenomenon and introduce a dormancy-guided reset mechanism to further enhance the efficacy of our method. Our proposed algorithm, ACE: Off-policy Actor-critic with Causality-aware Entropy regularization, demonstrates a substantial performance advantage across 29 diverse continuous control tasks spanning 7…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware System Performance and Reliability · Network Security and Intrusion Detection · Formal Methods in Verification
MethodsFocus
