SemanticFeels: Semantic Labeling during In-Hand Manipulation

TL;DR

SemanticFeels advances robotic in-hand manipulation by integrating semantic material labeling with neural implicit shape representations, combining vision and touch data for improved material recognition during manipulation.

Contribution

It introduces SemanticFeels, a novel framework that fuses tactile and visual data into neural implicit models for semantic labeling of objects in manipulation tasks.

Findings

Achieves 79.87% accuracy in material classification

Effectively integrates tactile and visual data for semantic labeling

Demonstrates high correspondence between predicted and actual materials

Abstract

As robots become increasingly integrated into everyday tasks, their ability to perceive both the shape and properties of objects during in-hand manipulation becomes critical for adaptive and intelligent behavior. We present SemanticFeels, an extension of the NeuralFeels framework that integrates semantic labeling with neural implicit shape representation, from vision and touch. To illustrate its application, we focus on material classification: high-resolution Digit tactile readings are processed by a fine-tuned EfficientNet-B0 convolutional neural network (CNN) to generate local material predictions, which are then embedded into an augmented signed distance field (SDF) network that jointly predicts geometry and continuous material regions. Experimental results show that the system achieves a high correspondence between predicted and actual materials on both single- and multi-material objects, with an average matching accuracy of 79.87% across multiple manipulation trials on a multi-material object.