MAGIC-VFM: Meta-learning Adaptation for Ground Interaction Control with Visual Foundation Models

TL;DR

This paper introduces MAGIC-VFM, a meta-learning approach that uses visual foundation models to rapidly adapt off-road vehicle control models to complex terrain interactions, improving performance in real-world tests.

Contribution

The paper presents a novel offline meta-learning method that leverages visual foundation models for terrain feature extraction and combines it with adaptive control for real-time adjustment.

Findings

Significant performance improvements over baseline adaptive controllers.

Robustness demonstrated across various terrains and disturbances.

Mathematical guarantees of stability and robustness.

Abstract

Control of off-road vehicles is challenging due to the complex dynamic interactions with the terrain. Accurate modeling of these interactions is important to optimize driving performance, but the relevant physical phenomena are too complex to model from first principles. Therefore, we present an offline meta-learning algorithm to construct a rapidly-tunable model of residual dynamics and disturbances. Our model processes terrain images into features using a visual foundation model (VFM), then maps these features and the vehicle state to an estimate of the current actuation matrix using a deep neural network (DNN). We then combine this model with composite adaptive control to modify the last layer of the DNN in real time, accounting for the remaining terrain interactions not captured during offline training. We provide mathematical guarantees of stability and robustness for our controller and demonstrate the effectiveness of our method through simulations and hardware experiments with a tracked vehicle and a car-like robot. We evaluate our method outdoors on different slopes with varying slippage and actuator degradation disturbances, and compare against an adaptive controller that does not use the VFM terrain features. We show significant improvement over the baseline in both hardware experimentation and simulation.