Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making

TL;DR

This paper introduces a contrastive learning approach to successor features that define a valid metric for temporal distances in stochastic environments, enabling better planning and faster learning.

Contribution

It demonstrates how contrastive successor features can form a triangle-inequality-satisfying temporal distance, improving generalization and efficiency in reinforcement learning.

Findings

Temporal distances satisfy the triangle inequality in stochastic settings.

The proposed method enables efficient estimation of temporal distances in high-dimensional spaces.

RL algorithms using these distances show improved generalization and learning speed.

Abstract

Temporal distances lie at the heart of many algorithms for planning, control, and reinforcement learning that involve reaching goals, allowing one to estimate the transit time between two states. However, prior attempts to define such temporal distances in stochastic settings have been stymied by an important limitation: these prior approaches do not satisfy the triangle inequality. This is not merely a definitional concern, but translates to an inability to generalize and find shortest paths. In this paper, we build on prior work in contrastive learning and quasimetrics to show how successor features learned by contrastive learning (after a change of variables) form a temporal distance that does satisfy the triangle inequality, even in stochastic settings. Importantly, this temporal distance is computationally efficient to estimate, even in high-dimensional and stochastic settings. Experiments in controlled settings and benchmark suites demonstrate that an RL algorithm based on these new temporal distances exhibits combinatorial generalization (i.e., "stitching") and can sometimes learn more quickly than prior methods, including those based on quasimetrics.