Context Representation via Action-Free Transformer encoder-decoder for Meta Reinforcement Learning

TL;DR

This paper introduces CRAFT, an action-free transformer-based model for meta reinforcement learning that improves task inference, generalization, and adaptation by decoupling belief updates from policy dependence.

Contribution

CRAFT is a novel belief model that infers task representations solely from states and rewards, enabling scalable, modular, and action-independent meta-RL.

Findings

CRAFT achieves faster adaptation in robotic tasks.

CRAFT improves generalization over baselines.

CRAFT enables more effective exploration.

Abstract

Reinforcement learning (RL) enables robots to operate in uncertain environments, but standard approaches often struggle with poor generalization to unseen tasks. Context-adaptive meta reinforcement learning addresses these limitations by conditioning on the task representation, yet they mostly rely on complete action information in the experience making task inference tightly coupled to a specific policy. This paper introduces Context Representation via Action Free Transformer encoder decoder (CRAFT), a belief model that infers task representations solely from sequences of states and rewards. By removing the dependence on actions, CRAFT decouples task inference from policy optimization, supports modular training, and leverages amortized variational inference for scalable belief updates. Built on a transformer encoder decoder with rotary positional embeddings, the model captures long range temporal dependencies and robustly encodes both parametric and non-parametric task variations. Experiments on the MetaWorld ML-10 robotic manipulation benchmark show that CRAFT achieves faster adaptation, improved generalization, and more effective exploration compared to context adaptive meta--RL baselines. These findings highlight the potential of action-free inference as a foundation for scalable RL in robotic control.