TransTouch: Learning Transparent Objects Depth Sensing Through Sparse Touches

TL;DR

This paper introduces TransTouch, a method that uses tactile feedback to fine-tune stereo depth networks for transparent objects, significantly improving real-world sensing accuracy with minimal data collection.

Contribution

It proposes a novel tactile-guided finetuning approach with an optimized probing strategy and confidence regularization for transparent object depth sensing.

Findings

Enhanced depth sensing accuracy for transparent objects

Effective tactile-guided finetuning reduces data requirements

Constructed a real-world dataset for evaluation

Abstract

Transparent objects are common in daily life. However, depth sensing for transparent objects remains a challenging problem. While learning-based methods can leverage shape priors to improve the sensing quality, the labor-intensive data collection in the real world and the sim-to-real domain gap restrict these methods' scalability. In this paper, we propose a method to finetune a stereo network with sparse depth labels automatically collected using a probing system with tactile feedback. We present a novel utility function to evaluate the benefit of touches. By approximating and optimizing the utility function, we can optimize the probing locations given a fixed touching budget to better improve the network's performance on real objects. We further combine tactile depth supervision with a confidence-based regularization to prevent over-fitting during finetuning. To evaluate the effectiveness of our method, we construct a real-world dataset including both diffuse and transparent objects. Experimental results on this dataset show that our method can significantly improve real-world depth sensing accuracy, especially for transparent objects.