DefVINS: Visual-Inertial Odometry for Deformable Scenes

TL;DR

DefVINS introduces a novel visual-inertial odometry method tailored for deformable scenes, combining scene deformation modeling with an observability analysis, and provides the first real-world benchmark for this challenging environment.

Contribution

It presents the first deformable scene visual-inertial odometry pipeline and a new benchmark dataset with ground-truth for evaluation.

Findings

DefVINS outperforms rigid and non-rigid baselines in deformable environments.

The observability analysis informs the use of IMU anchoring for better pose estimation.

Experimental results validate the effectiveness of the proposed approach.

Abstract

Deformable scenes violate the rigidity assumptions underpinning classical visual--inertial odometry (VIO), often leading to over-fitting to local non-rigid motion or to severe camera pose drift when deformation dominates visual parallax. In this paper, we introduce DefVINS, the first visual-inertial odometry pipeline designed to operate in deformable environments. Our approach models the odometry state by decomposing it into a rigid, IMU-anchored component and a non-rigid scene warp represented by an embedded deformation graph. As a second contribution, we present VIMandala, the first benchmark containing real images and ground-truth camera poses for visual-inertial odometry in deformable scenes. In addition, we augment the synthetic Drunkard's benchmark with simulated inertial measurements to further evaluate our pipeline under controlled conditions. We also provide an observability analysis of the visual-inertial deformable odometry problem, characterizing how inertial measurements constrain camera motion and render otherwise unobservable modes identifiable in the presence of deformation. This analysis motivates the use of IMU anchoring and leads to a conditioning-based activation strategy that avoids ill-posed updates under poor excitation. Experimental results on both the synthetic Drunkard's and our real VIMandala benchmarks show that DefVINS outperforms rigid visual--inertial and non-rigid visual odometry baselines. Our source code and data will be released upon acceptance.