Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning

TL;DR

This paper introduces a theoretically grounded counteractive reinforcement learning paradigm that enhances efficiency and scalability in high-dimensional MDPs, achieving faster training and better performance without extra computational costs.

Contribution

We propose a novel counteractive RL approach based on experience manipulation, providing a theoretical foundation and demonstrating significant empirical improvements in high-dimensional environments.

Findings

Achieves faster training with substantial sample efficiency.

Provides a theoretical basis for scalable RL in high-dimensional spaces.

Demonstrates significant performance gains in Arcade Learning Environment.

Abstract

Following the pivotal success of learning strategies to win at tasks, solely by interacting with an environment without any supervision, agents have gained the ability to make sequential decisions in complex MDPs. Yet, reinforcement learning policies face exponentially growing state spaces in high dimensional MDPs resulting in a dichotomy between computational complexity and policy success. In our paper we focus on the agent's interaction with the environment in a high-dimensional MDP during the learning phase and we introduce a theoretically-founded novel paradigm based on experiences obtained through counteractive actions. Our analysis and method provide a theoretical basis for efficient, effective, scalable and accelerated learning, and further comes with zero additional computational complexity while leading to significant acceleration in training. We conduct extensive experiments in the Arcade Learning Environment with high-dimensional state representation MDPs. The experimental results further verify our theoretical analysis, and our method achieves significant performance increase with substantial sample-efficiency in high-dimensional environments.