Zero-Shot Adaptation to Robot Structural Damage via Natural Language-Informed Kinodynamics Modeling

TL;DR

This paper introduces ZLIK, a zero-shot learning approach that uses natural language descriptions to adapt kinodynamic models of robots to structural damages, achieving significant error reduction and generalization across simulation and real-world scenarios.

Contribution

It presents a novel method that grounds damage descriptions in kinodynamic modeling using self-supervised learning, enabling zero-shot adaptation to various structural damages.

Findings

Achieves up to 81% reduction in kinodynamics error.

Generalizes across sim-to-real and scale gaps.

Effectively models diverse structural damages.

Abstract

High-performance autonomous mobile robots endure significant mechanical stress during in-the-wild operations, e.g., driving at high speeds or over rugged terrain. Although these platforms are engineered to withstand such conditions, mechanical degradation is inevitable. Structural damage manifests as consistent and notable changes in kinodynamic behavior compared to a healthy vehicle. Given the heterogeneous nature of structural failures, quantifying various damages to inform kinodynamics is challenging. We posit that natural language can describe and thus capture this variety of damages. Therefore, we propose Zero-shot Language Informed Kinodynamics (ZLIK), which employs self-supervised learning to ground semantic information of damage descriptions in kinodynamic behaviors to learn a forward kinodynamics model in a data-driven manner. Using the high-fidelity soft-body physics simulator BeamNG.tech, we collect data from a variety of structurally compromised vehicles. Our learned model achieves zero-shot adaptation to different damages with up to 81% reduction in kinodynamics error and generalizes across the sim-to-real and full-to-1/10$^{\text{th}}$ scale gaps.