Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer

TL;DR

Swim2Real introduces a vision-language model-based pipeline that calibrates a robotic fish simulator directly from swimming videos, effectively closing the sim-to-real gap without manual tuning and enabling zero-shot reinforcement learning transfer.

Contribution

This work presents the first VLM-guided, fully automated calibration method for aquatic robot simulators, eliminating manual stages and improving real-world policy transfer.

Findings

Achieves 43% lower MAE in velocity matching compared to previous methods.

Triples acceptance rate of parameter proposals using backtracking line search.

Enables zero-shot RL policy transfer to physical fish with 12% farther swimming distance.

Abstract

We present Swim2Real, a pipeline that calibrates a 16-parameter robotic fish simulator from swimming videos using vision-language model (VLM) feedback, requiring no hand-designed search stages. Calibrating soft aquatic robots is particularly challenging because nonlinear fluid-structure coupling makes the parameter landscape chaotic, simplified fluid models introduce a persistent sim-to-real gap, and controlled aquatic experiments are difficult to reproduce. Prior work on this platform required three manually tailored stages to handle this complexity. The VLM compares simulated and real videos and proposes parameter updates. A backtracking line search then validates each step size, tripling the accept rate from 14% to 42% by recovering proposals where the direction is correct but the magnitude is too large. Swim2Real calibrates all 16 parameters simultaneously, most closely matching real fish velocities across all motor frequencies (MAE = 7.4 mm/s, 43% lower than the next-best method), with zero outlier seeds across five runs. Motor commands from the trained policy transfer to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment. Downstream RL policies swim 12% farther than those from BayesOpt-calibrated simulators and 90% farther than CMA-ES. These results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification, a step toward automated, general-purpose simulator tuning for underwater robotics.