Reinforcement Learning via Auxiliary Task Distillation

TL;DR

AuxDistill introduces a novel reinforcement learning approach that distills auxiliary task behaviors to improve long-horizon robot control, achieving significant success in complex object rearrangement tasks without demonstrations.

Contribution

It presents AuxDistill, a new method for RL that leverages auxiliary task distillation to enhance learning in complex, multi-stage robotic tasks without pre-training or demonstrations.

Findings

Achieves 2.3x higher success rate than previous state-of-the-art in Habitat Object Rearrangement.

Learns pixels-to-actions policy for complex multi-stage tasks from environment rewards.

Outperforms methods using pre-trained skills and expert demonstrations.

Abstract

We present Reinforcement Learning via Auxiliary Task Distillation (AuxDistill), a new method that enables reinforcement learning (RL) to perform long-horizon robot control problems by distilling behaviors from auxiliary RL tasks. AuxDistill achieves this by concurrently carrying out multi-task RL with auxiliary tasks, which are easier to learn and relevant to the main task. A weighted distillation loss transfers behaviors from these auxiliary tasks to solve the main task. We demonstrate that AuxDistill can learn a pixels-to-actions policy for a challenging multi-stage embodied object rearrangement task from the environment reward without demonstrations, a learning curriculum, or pre-trained skills. AuxDistill achieves $2.3 \times$ higher success than the previous state-of-the-art baseline in the Habitat Object Rearrangement benchmark and outperforms methods that use pre-trained skills and expert demonstrations.