Build generally reusable agent-environment interaction models

TL;DR

This paper introduces a method for pre-training a general agent-environment interaction model using domain-invariant successor features and behavior prototypes, enabling effective transfer to new tasks with environmental and objective changes.

Contribution

It proposes a novel approach combining successor features, embodied set structures, and projected Bellman updates to create reusable, adaptable agent models for diverse downstream tasks.

Findings

Pre-trained models can handle unseen task objectives.

The approach adapts to environmental dynamics changes.

It maintains knowledge across different sensor modalities.

Abstract

This paper tackles the problem of how to pre-train a model and make it generally reusable backbones for downstream task learning. In pre-training, we propose a method that builds an agent-environment interaction model by learning domain invariant successor features from the agent's vast experiences covering various tasks, then discretize them into behavior prototypes which result in an embodied set structure. To make the model generally reusable for downstream task learning, we propose (1) embodied feature projection that retains previous knowledge by projecting the new task's observation-action pair to the embodied set structure and (2) projected Bellman updates which add learning plasticity for the new task setting. We provide preliminary results that show downstream task learning based on a pre-trained embodied set structure can handle unseen changes in task objectives, environmental dynamics and sensor modalities.