RoboHorizon: An LLM-Assisted Multi-View World Model for Long-Horizon Robotic Manipulation

TL;DR

RoboHorizon introduces an LLM-assisted multi-view world model that enhances long-horizon robotic manipulation by improving task recognition, perception, and planning, leading to significant performance gains in benchmark environments.

Contribution

The paper presents RoboHorizon, a novel multi-view world model leveraging LLMs and keyframe discovery to improve long-horizon robotic manipulation tasks.

Findings

Achieved 23.35% higher success rate on RLBench short-horizon tasks.

Achieved 29.23% higher success rate on long-horizon and furniture assembly tasks.

Outperforms state-of-the-art visual model-based RL methods.

Abstract

Efficient control in long-horizon robotic manipulation is challenging due to complex representation and policy learning requirements. Model-based visual reinforcement learning (RL) has shown great potential in addressing these challenges but still faces notable limitations, particularly in handling sparse rewards and complex visual features in long-horizon environments. To address these limitations, we propose the Recognize-Sense-Plan-Act (RSPA) pipeline for long-horizon tasks and further introduce RoboHorizon, an LLM-assisted multi-view world model tailored for long-horizon robotic manipulation. In RoboHorizon, pre-trained LLMs generate dense reward structures for multi-stage sub-tasks based on task language instructions, enabling robots to better recognize long-horizon tasks. Keyframe discovery is then integrated into the multi-view masked autoencoder (MAE) architecture to enhance the robot's ability to sense critical task sequences, strengthening its multi-stage perception of long-horizon processes. Leveraging these dense rewards and multi-view representations, a robotic world model is constructed to efficiently plan long-horizon tasks, enabling the robot to reliably act through RL algorithms. Experiments on two representative benchmarks, RLBench and FurnitureBench, show that RoboHorizon outperforms state-of-the-art visual model-based RL methods, achieving a 23.35% improvement in task success rates on RLBench's 4 short-horizon tasks and a 29.23% improvement on 6 long-horizon tasks from RLBench and 3 furniture assembly tasks from FurnitureBench.