Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learning

TL;DR

This paper introduces Metric Residual Networks (MRN), a novel neural architecture for goal-conditioned reinforcement learning that leverages metric properties to significantly improve sample efficiency across various benchmarks.

Contribution

The paper proposes MRN, a new neural network architecture that decomposes the action-value function to better satisfy metric properties, enhancing sample efficiency in GCRL tasks.

Findings

MRN outperforms existing architectures in 12 benchmark environments.

MRN achieves higher sample efficiency than state-of-the-art methods.

The architecture is theoretically grounded in metric properties of the value function.

Abstract

Goal-conditioned reinforcement learning (GCRL) has a wide range of potential real-world applications, including manipulation and navigation problems in robotics. Especially in such robotics tasks, sample efficiency is of the utmost importance for GCRL since, by default, the agent is only rewarded when it reaches its goal. While several methods have been proposed to improve the sample efficiency of GCRL, one relatively under-studied approach is the design of neural architectures to support sample efficiency. In this work, we introduce a novel neural architecture for GCRL that achieves significantly better sample efficiency than the commonly-used monolithic network architecture. The key insight is that the optimal action-value function Q^*(s, a, g) must satisfy the triangle inequality in a specific sense. Furthermore, we introduce the metric residual network (MRN) that deliberately decomposes the action-value function Q(s,a,g) into the negated summation of a metric plus a residual asymmetric component. MRN provably approximates any optimal action-value function Q^*(s,a,g), thus making it a fitting neural architecture for GCRL. We conduct comprehensive experiments across 12 standard benchmark environments in GCRL. The empirical results demonstrate that MRN uniformly outperforms other state-of-the-art GCRL neural architectures in terms of sample efficiency.