TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality

TL;DR

This paper introduces TACT, a deep imitation learning approach enabling humanoid robots to perform stable whole-body contact manipulation by integrating tactile, visual, and joint data, demonstrated on a life-size robot.

Contribution

The paper presents a novel tactile-modality extended ACT (TACT) policy that combines multiple sensor inputs for robust whole-body manipulation in humanoid robots.

Findings

Integrating tactile and vision data improves manipulation robustness.

The policy enables a humanoid robot to maintain balance during contact tasks.

Experimental results validate the effectiveness of the approach on a real robot.

Abstract

Manipulation with whole-body contact by humanoid robots offers distinct advantages, including enhanced stability and reduced load. On the other hand, we need to address challenges such as the increased computational cost of motion generation and the difficulty of measuring broad-area contact. We therefore have developed a humanoid control system that allows a humanoid robot equipped with tactile sensors on its upper body to learn a policy for whole-body manipulation through imitation learning based on human teleoperation data. This policy, named tactile-modality extended ACT (TACT), has a feature to take multiple sensor modalities as input, including joint position, vision, and tactile measurements. Furthermore, by integrating this policy with retargeting and locomotion control based on a biped model, we demonstrate that the life-size humanoid robot RHP7 Kaleido is capable of achieving whole-body contact manipulation while maintaining balance and walking. Through detailed experimental verification, we show that inputting both vision and tactile modalities into the policy contributes to improving the robustness of manipulation involving broad and delicate contact.