AME-2: Agile and Generalized Legged Locomotion via Attention-Based Neural Map Encoding

TL;DR

This paper presents AME-2, a reinforcement learning framework that combines attention-based neural map encoding and uncertainty-aware terrain mapping to enable agile, generalized legged locomotion across diverse terrains, including occlusions.

Contribution

Introduction of AME-2, a unified RL framework with an attention-based map encoder and a novel terrain mapping pipeline for robust, interpretable, and generalized legged robot control.

Findings

Controllers exhibit strong agility and generalization in simulation.

Effective real-world transfer demonstrated on quadruped and biped robots.

Robustness to occlusions and noisy terrain data achieved.

Abstract

Achieving agile and generalized legged locomotion across terrains requires tight integration of perception and control, especially under occlusions and sparse footholds. Existing methods have demonstrated agility on parkour courses but often rely on end-to-end sensorimotor models with limited generalization and interpretability. By contrast, methods targeting generalized locomotion typically exhibit limited agility and struggle with visual occlusions. We introduce AME-2, a unified reinforcement learning (RL) framework for agile and generalized locomotion that incorporates a novel attention-based map encoder in the control policy. This encoder extracts local and global mapping features and uses attention mechanisms to focus on salient regions, producing an interpretable and generalized embedding for RL-based control. We further propose a learning-based mapping pipeline that provides fast, uncertainty-aware terrain representations robust to noise and occlusions, serving as policy inputs. It uses neural networks to convert depth observations into local elevations with uncertainties, and fuses them with odometry. The pipeline also integrates with parallel simulation so that we can train controllers with online mapping, aiding sim-to-real transfer. We validate AME-2 with the proposed mapping pipeline on a quadruped and a biped robot, and the resulting controllers demonstrate strong agility and generalization to unseen terrains in simulation and in real-world experiments.