Exploration by Running Away from the Past
Paul-Antoine Le Tolguenec, Yann Besse, Florent Teichteil-Koenigsbuch,, Dennis G. Wilson, Emmanuel Rachelson
TL;DR
This paper introduces RAMP, a reinforcement learning exploration method that maximizes the divergence from past behaviors using information theory, leading to improved exploration in complex environments.
Contribution
The paper proposes RAMP, a novel exploration strategy based on maximizing divergence from past behaviors, with analysis of divergence measures and demonstrated effectiveness in various tasks.
Findings
RAMP effectively explores mazes and robotic tasks.
Divergence measures influence exploration quality.
Active distancing from past behaviors enhances exploration.
Abstract
The ability to explore efficiently and effectively is a central challenge of reinforcement learning. In this work, we consider exploration through the lens of information theory. Specifically, we cast exploration as a problem of maximizing the Shannon entropy of the state occupation measure. This is done by maximizing a sequence of divergences between distributions representing an agent's past behavior and its current behavior. Intuitively, this encourages the agent to explore new behaviors that are distinct from past behaviors. Hence, we call our method RAMP, for ``unning way fro the ast.'' A fundamental question of this method is the quantification of the distribution change over time. We consider both the Kullback-Leibler divergence and the Wasserstein distance to quantify divergence between successive state occupation measures, and…
Peer Reviews
Decision·Submitted to ICLR 2025
Though the problem of state space exploration is very extensively covered in the RL literature, the proposed RAMP method provides what appears to be a novel approach to accelerating state space coverage. Due to its strategy of choosing policies maximizing divergence of state space coverage from that achieved by previous policies, it makes sense that RAMP will be more effective at rapidly exploring the state space than existing unsupervised RL methods (e.g., APT, SMM, Proto-RL) that simply maximi
Despite the strengths discussed above, I have concerns about the experimental evaluation and theoretical results: 1. Most importantly, the "state coverage" performance metric upon which the comparisons of Sections 5.2 and A.1 rely is insufficiently justified as a good proxy for measuring exploration and for making fair comparisons between the algorithms considered. As described in the third paragraph of Sec. 5.2, this metric is obtained by discretizing the space of Euclidean (x-y or x-y-z) coord
1. The problem addressed is important to the community. 2. The new objective function is theoretically motivated and provides new insights to compute good exploration policies.
1. In Section 2, different justifications for introducing the learning objective pursued by the agent are wrong or weak in several aspects: a. The justification line 108 for going from equation (1) to equation (2) is in my opinion wrong. Using the entropy of the policy as proxy to the entropy of the state distribution is a huge approximation. Maximizing the entropy of the policy does not provide a good state coverage in general nor in most practical cases. Note that if it was sufficient to maxi
The strength of the paper lies in a fairly clear presentation of the motivation and methodology. The idea of "running away from the past" is not strictly novel but the paper proposes an algorithmically viable way to instantiate such an idea. The paper presents a fairly clear math formulation and has carried out ablations on choices of the algorithmic designs. The experimental ablation also seems fairly comprehensive.
The idea of "running away from the past" is not strictly novel. From a theoretical standpoint, running away from old trajectories might not always be optimal and it is not clear theoretically what is gained by adopting such an approach. From an empirical standpoint, the ablations are carried out on the continuous control tasks, most of which do not seem to require extensive exploration to solve. It is not very clear if the claimed gains are really due to the exploration bonus, or some other unkn
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Robot Manipulation and Learning · Embodied and Extended Cognition