Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective

TL;DR

This paper introduces LIBERO-Mem, a challenging object-centric manipulation task suite, and proposes Embodied-SlotSSM, a scalable vision-language-action model that improves temporal reasoning for robotic manipulation in complex, non-Markovian environments.

Contribution

It presents LIBERO-Mem for testing non-Markovian robotic manipulation and introduces Embodied-SlotSSM, a scalable, slot-centric VLA framework for enhanced temporal reasoning.

Findings

Embodied-SlotSSM outperforms baseline models on LIBERO-Mem.

The framework maintains consistent object identities over time.

It enables better action prediction in complex manipulation tasks.

Abstract

As embodied agents operate in increasingly complex environments, the ability to perceive, track, and reason about individual object instances over time becomes essential, especially in tasks requiring sequenced interactions with visually similar objects. In these non-Markovian settings, key decision cues are often hidden in object-specific histories rather than the current scene. Without persistent memory of prior interactions (what has been interacted with, where it has been, or how it has changed) visuomotor policies may fail, repeat past actions, or overlook completed ones. To surface this challenge, we introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability. It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame. However, vision-language-action (VLA) models often struggle in such settings, with token scaling quickly becoming intractable even for tasks spanning just a few hundred frames. We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability. It maintains spatio-temporally consistent slot identities and leverages them through two mechanisms: (1) slot-state-space modeling for reconstructing short-term history, and (2) a relational encoder to align the input tokens with action decoding. Together, these components enable temporally grounded, context-aware action prediction. Experiments show Embodied-SlotSSM's baseline performance on LIBERO-Mem and general tasks, offering a scalable solution for non-Markovian reasoning in object-centric robotic policies.