SigLoMa: Learning Open-World Quadrupedal Loco-Manipulation from Ego-Centric Vision

TL;DR

SigLoMa introduces a fully onboard, ego-centric vision-based quadrupedal loco-manipulation system that overcomes traditional limitations using Sigma Points, an ego-centric Kalman Filter, and active sampling, enabling real-world dynamic tasks.

Contribution

The paper presents SigLoMa, a novel onboard system with Sigma Points and ego-centric Kalman filtering that improves open-world quadrupedal loco-manipulation from vision.

Findings

Successfully performs dynamic loco-manipulation tasks in real-world settings.

Achieves performance comparable to expert human teleoperation.

Operates effectively with only a 5Hz perception update rate.

Abstract

Designing an open-world quadrupedal loco-manipulation system is highly challenging. Traditional reinforcement learning frameworks utilizing exteroception often suffer from extreme sample inefficiency and massive sim-to-real gaps. Furthermore, the inherent latency of visual tracking fundamentally conflicts with the high-frequency demands of precise floating-base control. Consequently, existing systems lean heavily on expensive external motion capture and off-board computation. To eliminate these dependencies, we present SigLoMa, a fully onboard, ego-centric vision-based pick-and-place framework. At the core of SigLoMa is the introduction of Sigma Points, a lightweight geometric representation for exteroception that guarantees high scalability and native sim-to-real alignment. To bridge the frequency divide between slow perception and fast control, we design an ego-centric Kalman Filter to provide robust, high-rate state estimation. On the learning front, we alleviate sample inefficiency via an Active Sampling Curriculum guided by Hint Poses, and tackle the robot's structural visual blind spots using temporal encoding coupled with simulated random-walk drift. Real-world experiments validate that, relying solely on a 5Hz (200 ms latency) open-vocabulary detector, SigLoMa successfully executes dynamic loco-manipulation across multiple tasks, achieving performance comparable to expert human teleoperation.