Preferenced Oracle Guided Multi-mode Policies for Dynamic Bipedal Loco-Manipulation

TL;DR

This paper introduces OGMP, a unified policy learning approach for dynamic bipedal loco-manipulation tasks, enabling smooth transitions between modes like running, dribbling, and kicking, demonstrated across various robots.

Contribution

It proposes a novel multi-mode policy framework guided by hybrid automatons as oracles, with a preference reward to learn seamless mode transitions for complex loco-manipulation tasks.

Findings

Successful application to soccer and box-moving tasks

Single policy effective across different robot morphologies

Enhanced performance through preference-guided mode transitions

Abstract

Dynamic loco-manipulation calls for effective whole-body control and contact-rich interactions with the object and the environment. Existing learning-based control synthesis relies on training low-level skill policies and explicitly switching with a high-level policy or a hand-designed finite state machine, leading to quasi-static behaviors. In contrast, dynamic tasks such as soccer require the robot to run towards the ball, decelerate to an optimal approach to dribble, and eventually kick a goal - a continuum of smooth motion. To this end, we propose Preferenced Oracle Guided Multi-mode Policies (OGMP) to learn a single policy mastering all the required modes and preferred sequence of transitions to solve uni-object loco-manipulation tasks. We design hybrid automatons as oracles to generate references with continuous dynamics and discrete mode jumps to perform a guided policy optimization through bounded exploration. To enforce learning a desired sequence of mode transitions, we present a task-agnostic preference reward that enhances performance. The proposed approach demonstrates successful loco-manipulation for tasks like soccer and moving boxes omnidirectionally through whole-body control. In soccer, a single policy learns to optimally reach the ball, transition to contact-rich dribbling, and execute successful goal kicks and ball stops. Leveraging the oracle's abstraction, we solve each loco-manipulation task on robots with varying morphologies, including HECTOR V1, Berkeley Humanoid, Unitree G1, and H1, using the same reward definition and weights.