In-Context Planning with Latent Temporal Abstractions

TL;DR

I-TAP is an offline RL framework that combines in-context adaptation with online planning by learning a discrete latent space of temporal abstractions, enabling efficient decision-making in complex, partially observable environments.

Contribution

It introduces a novel latent temporal abstraction space learned from offline data, facilitating in-context planning and adaptation without gradient updates during testing.

Findings

Outperforms strong offline baselines in MuJoCo and manipulation tasks.

Handles stochastic dynamics and partial observability effectively.

Enables efficient planning in high-dimensional, real-world scenarios.

Abstract

Planning-based reinforcement learning for continuous control is bottlenecked by two practical issues: planning at primitive time scales leads to prohibitive branching and long horizons, while real environments are frequently partially observable and exhibit regime shifts that invalidate stationary, fully observed dynamics assumptions. We introduce I-TAP (In-Context Latent Temporal-Abstraction Planner), an offline RL framework that unifies in-context adaptation with online planning in a learned discrete temporal-abstraction space. From offline trajectories, I-TAP learns an observation-conditioned residual-quantization VAE that compresses each observation-macro-action segment into a coarse-to-fine stack of discrete residual tokens, and a temporal Transformer that autoregressively predicts these token stacks from a short recent history. The resulting sequence model acts simultaneously as a context-conditioned prior over abstract actions and a latent dynamics model. At test time, I-TAP performs Monte Carlo Tree Search directly in token space, using short histories for implicit adaptation without gradient update, and decodes selected token stacks into executable actions. Across deterministic MuJoCo, stochastic MuJoCo with per-episode latent dynamics regimes, and high-dimensional Adroit manipulation, including partially observable variants, I-TAP consistently matches or outperforms strong model-free and model-based offline baselines, demonstrating efficient and robust in-context planning under stochastic dynamics and partial observability.