TMR-VLA:Vision-Language-Action Model for Magnetic Motion Control of Tri-leg Silicone-based Soft Robot

TL;DR

This paper introduces TMR-VLA, a multi-modal system enabling a magnetically driven soft robot to perform diverse, shape-dependent motions controlled via natural language commands, enhancing navigation in in-vivo environments.

Contribution

The paper presents TMR-VLA, an innovative end-to-end system that links language commands to low-level voltage control for a multi-legged soft robot, enabling flexible, shape-dependent motions.

Findings

Achieved 74% success rate in motion prediction

Demonstrated effective hybrid motion control via language commands

Enabled flexible gait patterns in soft robots

Abstract

In-vivo environments, magnetically actuated soft robots offer advantages such as wireless operation and precise control, showing promising potential for painless detection and therapeutic procedures. We developed a trileg magnetically driven soft robot (TMR) whose multi-legged design enables more flexible gaits and diverse motion patterns. For the silicone made of reconfigurable soft robots, its navigation ability can be separated into sequential motions, namely squatting, rotation, lifting a leg, walking and so on. Its motion and behavior depend on its bending shapes. To bridge motion type description and specific low-level voltage control, we introduced TMR-VLA, an end-to-end multi-modal system for a trileg magnetic soft robot capable of performing hybrid motion types, which is promising for developing a navigation ability by adapting its shape to language-constrained motion types. The TMR-VLA deploys embodied endoluminal localization ability from EndoVLA, and fuses sequential frames and natural language commands as input. Low-level voltage output is generated based on the current observation state and specific motion type description. The result shows the TMR-VLA can predict how the voltage applied to TMR will change the dynamics of a silicon-made soft robot. The TMR-VLA reached a 74% average success rate.