Learning Agile Striker Skills for Humanoid Soccer Robots from Noisy Sensory Input

TL;DR

This paper introduces a reinforcement learning system enabling humanoid robots to perform robust, adaptable ball-kicking skills under noisy sensory input, bridging the gap between simulation and real-world performance.

Contribution

It extends a teacher-student RL framework with multiple training stages, realistic noise modeling, and online adaptation to improve robustness and sim-to-real transfer for humanoid soccer skills.

Findings

Robust kicking accuracy achieved in both simulation and real robots.

The system maintains performance under noisy sensory conditions and external perturbations.

A new benchmark task for visuomotor skill learning in humanoid control is established.

Abstract

Learning fast and robust ball-kicking skills is a critical capability for humanoid soccer robots, yet it remains a challenging problem due to the need for rapid leg swings, postural stability on a single support foot, and robustness under noisy sensory input and external perturbations (e.g., opponents). This paper presents a reinforcement learning (RL)-based system that enables humanoid robots to execute robust continual ball-kicking with adaptability to different ball-goal configurations. The system extends a typical teacher-student training framework -- in which a "teacher" policy is trained with ground truth state information and the "student" learns to mimic it with noisy, imperfect sensing -- by including four training stages: (1) long-distance ball chasing (teacher); (2) directional kicking (teacher); (3) teacher policy distillation (student); and (4) student adaptation and refinement (student). Key design elements -- including tailored reward functions, realistic noise modeling, and online constrained RL for adaptation and refinement -- are critical for closing the sim-to-real gap and sustaining performance under perceptual uncertainty. Extensive evaluations in both simulation and on a real robot demonstrate strong kicking accuracy and goal-scoring success across diverse ball-goal configurations. Ablation studies further highlight the necessity of the constrained RL, noise modeling, and the adaptation stage. This work presents a system for learning robust continual humanoid ball-kicking under imperfect perception, establishing a benchmark task for visuomotor skill learning in humanoid whole-body control.