LoLA: Long Horizon Latent Action Learning for General Robot Manipulation

TL;DR

LoLA introduces a novel framework for long-horizon, language-guided robot manipulation that integrates multi-view observations and robot proprioception to improve multi-step reasoning and action generation.

Contribution

It proposes a state-aware latent re-representation module that grounds visual and language inputs in physical robot states, enhancing long-horizon manipulation capabilities.

Findings

LoLA outperforms prior methods on simulation benchmarks.

LoLA achieves significant improvements on real-world robotic tasks.

The approach effectively integrates multi-view observations and proprioception.

Abstract

The capability of performing long-horizon, language-guided robotic manipulation tasks critically relies on leveraging historical information and generating coherent action sequences. However, such capabilities are often overlooked by existing Vision-Language-Action (VLA) models. To solve this challenge, we propose LoLA (Long Horizon Latent Action Learning), a framework designed for robot manipulation that integrates long-term multi-view observations and robot proprioception to enable multi-step reasoning and action generation. We first employ Vision-Language Models to encode rich contextual features from historical sequences and multi-view observations. We further introduces a key module, State-Aware Latent Re-representation, which transforms visual inputs and language commands into actionable robot motion space. Unlike existing VLA approaches that merely concatenate robot proprioception (e.g., joint angles) with VL embeddings, this module leverages such robot states to explicitly ground VL representations in physical scale through a learnable "embodiment-anchored" latent space. We trained LoLA on diverse robotic pre-training datasets and conducted extensive evaluations on simulation benchmarks (SIMPLER and LIBERO), as well as two real-world tasks on Franka and Bi-Manual Aloha robots. Results show that LoLA significantly outperforms prior state-of-the-art methods (e.g., pi0), particularly in long-horizon manipulation tasks.