The Termination Critic
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos,, Doina Precup

TL;DR
This paper introduces a novel approach to learning behavioral abstractions in reinforcement learning by focusing on the termination condition's information-theoretic properties, leading to more meaningful options.
Contribution
It proposes a new algorithm that optimizes option terminations based on compressibility, using an information-theoretic perspective and a critic for the transition model.
Findings
Options learned are non-trivial and meaningful.
The approach improves learning and planning efficiency.
The method offers a new perspective on option termination criteria.
Abstract
In this work, we consider the problem of autonomously discovering behavioral abstractions, or options, for reinforcement learning agents. We propose an algorithm that focuses on the termination condition, as opposed to -- as is common -- the policy. The termination condition is usually trained to optimize a control objective: an option ought to terminate if another has better value. We offer a different, information-theoretic perspective, and propose that terminations should focus instead on the compressibility of the option's encoding -- arguably a key reason for using abstractions. To achieve this algorithmically, we leverage the classical options framework, and learn the option transition model as a "critic" for the termination condition. Using this model, we derive gradients that optimize the desired criteria. We show that the resulting options are non-trivial, intuitively…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Adversarial Robustness in Machine Learning
