Rethinking Causality-driven Robot Tool Segmentation with Temporal Constraints

TL;DR

This paper introduces TC-CaRTS, a novel method that leverages temporal constraints to improve robot tool segmentation in videos, achieving faster convergence and better domain generalization over existing approaches.

Contribution

The paper proposes a new temporal causal model and architecture, TC-CaRTS, with three modules that enhance robot tool segmentation by utilizing temporal information.

Findings

TC-CaRTS converges faster than CaRTS, requiring fewer iterations.

TC-CaRTS achieves equal or better segmentation performance across different domains.

All three proposed modules are effective in improving segmentation results.

Abstract

Purpose: Vision-based robot tool segmentation plays a fundamental role in surgical robots and downstream tasks. CaRTS, based on a complementary causal model, has shown promising performance in unseen counterfactual surgical environments in the presence of smoke, blood, etc. However, CaRTS requires over 30 iterations of optimization to converge for a single image due to limited observability. Method: To address the above limitations, we take temporal relation into consideration and propose a temporal causal model for robot tool segmentation on video sequences. We design an architecture named Temporally Constrained CaRTS (TC-CaRTS). TC-CaRTS has three novel modules to complement CaRTS - temporal optimization pipeline, kinematics correction network, and spatial-temporal regularization. Results: Experiment results show that TC-CaRTS requires much fewer iterations to achieve the same or better performance as CaRTS. TC- CaRTS also has the same or better performance in different domains compared to CaRTS. All three modules are proven to be effective. Conclusion: We propose TC-CaRTS, which takes advantage of temporal constraints as additional observability. We show that TC-CaRTS outperforms prior work in the robot tool segmentation task with improved convergence speed on test datasets from different domains.