Continual Vision-based Reinforcement Learning with Group Symmetries

TL;DR

This paper introduces COVERS, a continual vision-based reinforcement learning method that leverages group symmetries to improve sample efficiency and generalization by recognizing task invariances under transformations.

Contribution

COVERS is the first method to incorporate group symmetries into continual RL with visual inputs, enabling task grouping and improved generalization.

Findings

COVERS accurately groups tasks based on invariances.

It outperforms existing methods in generalization on manipulation tasks.

Demonstrates effectiveness on both simulation and real robots.

Abstract

Continual reinforcement learning aims to sequentially learn a variety of tasks, retaining the ability to perform previously encountered tasks while simultaneously developing new policies for novel tasks. However, current continual RL approaches overlook the fact that certain tasks are identical under basic group operations like rotations or translations, especially with visual inputs. They may unnecessarily learn and maintain a new policy for each similar task, leading to poor sample efficiency and weak generalization capability. To address this, we introduce a unique Continual Vision-based Reinforcement Learning method that recognizes Group Symmetries, called COVERS, cultivating a policy for each group of equivalent tasks rather than individual tasks. COVERS employs a proximal policy optimization-based RL algorithm with an equivariant feature extractor and a novel task grouping mechanism that relies on the extracted invariant features. We evaluate COVERS on sequences of table-top manipulation tasks that incorporate image observations and robot proprioceptive information in both simulations and on real robot platforms. Our results show that COVERS accurately assigns tasks to their respective groups and significantly outperforms existing methods in terms of generalization capability.