AFT: Appearance-Based Feature Tracking for Markerless and Training-Free Shape Reconstruction of Soft Robots

TL;DR

This paper introduces a vision-based, markerless, and training-free method for real-time shape reconstruction of soft robots using natural surface features, enhancing practicality and robustness in diverse environments.

Contribution

It presents a novel hierarchical matching framework that leverages surface appearance as implicit markers, eliminating the need for complex setups or training datasets.

Findings

Achieves 2.6% tip error in real-time shape tracking

Demonstrates robustness to occlusions and viewpoint changes

Enables stable closed-loop control in practical tasks

Abstract

Accurate shape reconstruction is essential for precise control and reliable operation of soft robots. Compared to sensor-based approaches, vision-based methods offer advantages in cost, simplicity, and ease of deployment. However, existing vision-based methods often rely on complex camera setups, specific backgrounds, or large-scale training datasets, limiting their practicality in real-world scenarios. In this work, we propose a vision-based, markerless, and training-free framework for soft robot shape reconstruction that directly leverages the robot's natural surface appearance. These surface features act as implicit visual markers, enabling a hierarchical matching strategy that decouples local partition alignment from global kinematic optimization. Requiring only an initial 3D reconstruction and kinematic alignment, our method achieves real-time shape tracking across diverse environments while maintaining robustness to occlusions and variations in camera viewpoints. Experimental validation on a continuum soft robot demonstrates an average tip error of 2.6% during real-time operation, as well as stable performance in practical closed-loop control tasks. These results highlight the potential of the proposed approach for reliable, low-cost deployment in dynamic real-world settings.